Tag Archives: JSON.NET

Manually Parsing a Json String Using System.Text.Json

In this post from 2016 I gave some details as to how you could manually parse a JSON string using Newtonsoft JSON.NET. What I don’t think I covered in that post, was why you may wish to do that, nor did I cover how you could do that using System.Text.Json (although since the library was only introduced in .Net Core 3 – circa 2019, that part couldn’t really be helped!)

Why?

Let’s start with why you would want to parse a JSON string manually – I mean, the serialisation functions in, pretty much, any JSON library are really good.

The problem is coupling: by using serialisation, you’re coupling your data to a given shape, and very tightly coupling it to that shape, too. So much so, that a common pattern if you’re passing data between two services, and using serialisation, it to share the model class between the services. This sounds quite innocuous at first glance, but let’s consider a few factors (I’m assuming we’re talking exclusively about .Net, but I imagine the points are valid outside of that, too):

1. Why are you serialising and de-serialising the data to begin with?
2. Single Responsibility Principle.
3. Microservices.

Let’s start with (1). Your answer may be different, but typically, if you’re passing data as a string, it’s because you’re trying to remove a dependency to a given complex type. After all, a string can be passed anywhere: an API call, a message broker, even held in a database.

What has this got to do with the SRP (2)? Well, the SRP is typically used to describe the reason that a module has to change (admittedly it is slightly mis-named). Let’s see how the two modules may interact:

Now, let’s look at the interaction with a common module:

As you can see, they both have a dependency on a single external (external to the service) dependency. If the CustomerModel changes, then both services may also need to change, but they also need to change for alterations for business rules that relate to the module itself: so they now have two reasons to change.

Of course, you don’t have to have a common dependency like this; you could structure your system like this:

However, you don’t solve your problem – in fact, you arguably make it worse: if you change CustomerModel referenced by Service 1 you potentially break Service 2, so you now need to change CustomerModel referenced by Service 2, and Service 2!

Just to clarify what I’m saying here: there may be valid reasons for both of these designs – but if you use them, then be aware that you’re coupling the services to each other; which brings us to point (3): if you’re trying to create a Service Oriented Architecture of any description, then this kind of coupling may hinder your efforts.

The Solution

A quick caveat here: whatever you do in a system, the parts of that system will be coupled to some extent. For example, if you have built a Microservice Architecture where your system is running a nuclear reactor, and then you decide to change one of the services from monitoring the cooling tower to, instead, mine bit-coins, you’re going to find that there is an element of coupling. Some of that will be business coupling (i.e. the cooling tower may overheat), but some will be technical – a downstream service may depend on the monitoring service to assert that something is safe.

Apologies, I know absolutely nothing about nuclear reactors; or, for that matter, about mining bit-coin!

All that said, if you manually parse the JSON that’s received, you remove some dependency on the shape of the data.

The following reads a JSON document, and iterates through an array:

            using var doc = JsonDocument.Parse(json);
            var element = doc.RootElement;

            foreach (var eachElement in element.EnumerateArray())
            {
                string name = eachElement.GetProperty("Name").GetString();
                decimal someFigure = eachElement.GetProperty("SomeFigure").GetDecimal();

                if (someFigure > 500)
                {
                    Console.WriteLine($"{name} has more than 500!");
                }
            }

As you can see, if the property name of SomeFigure changed, the code would break; however, there may be a dozen more fields in each element that could change, and we wouldn’t care.

Seriliasing Interfaces in JSON (or using a JsonConverter in JSON.NET)

Imagine that you have the following interface:

    public interface IProduct
    {
        int Id { get; set; }
        decimal UnitPrice { get; set; }
    }

This is an interface, and so may have a number of implementations; however, we know that every implementation will contain at least 2 fields, and what type they will be. If we wanted to serialise this, we’d probably write something like this:

        private static string SerialiseProduct(IProduct product)
        {
            string json = JsonConvert.SerializeObject(product);
            return json;
        }

If you were to call this from a console app, it would work fine:

        static void Main(string[] args)
        {
            IProduct product = new Product()
            {
                Id = 1,
                UnitPrice = 12.3m
            };

            string json = SerialiseProduct(product);
            Console.WriteLine(json);

Okay, so far so good. Now, let’s deserialise:

        private static IProduct DeserialiseProduct(string json)
        {
            IProduct product = JsonConvert.DeserializeObject<IProduct>(json);

            return product;
        }

And let’s call it:

        static void Main(string[] args)
        {
            IProduct product = new Product()
            {
                Id = 1,
                UnitPrice = 12.3m
            };

            string json = SerialiseProduct(product);
            Console.WriteLine(json);

            IProduct product2 = DeserialiseProduct(json);
            Console.WriteLine(product2.Id);
            
            Console.ReadLine();

        }

So, that runs fine:

Newtonsoft.Json.JsonSerializationException: ‘Could not create an instance of type SerialiseInterfaceJsonNet.IProduct. Type is an interface or abstract class and cannot be instantiated.

No.

Why?

The reason is that you can’t create an interface; for example:

That doesn’t even compile, but effectively, that’s what’s happening behind the scenes.

Converters

Json.Net allows the use of something called a converter. What that means is that I can inject functionality into the deserialisation process that tells Json.Net what to do with this interface. Here’s a possible converter for our class:

    class ProductConverter : JsonConverter
    {
        public override bool CanConvert(Type objectType)
        {
            return (objectType == typeof(IProduct));
        }

        public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
        {
            return serializer.Deserialize(reader, typeof(Product));
        }

        public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
        {
            serializer.Serialize(writer, value, typeof(Product));
        }
    }

It’s a relatively simple interface, you tell it how to identify your class, and then how to read and write the Json.

Finally, you just need to tell the converter to use this:

        private static IProduct DeserialiseProduct(string json)
        {
            var settings = new JsonSerializerSettings();
            settings.Converters.Add(new ProductConverter());

            IProduct product = JsonConvert.DeserializeObject<IProduct>(json, settings);

            return product;
        }

By using the settings parameter.

References

http://www.jerriepelser.com/blog/custom-converters-in-json-net-case-study-1/

Manually Parsing a JSON String Using JSON.NET

How to manually parse a JSON string using JSON.NET.

Disclaimer

If you jump straight to the references, you will find a very similar set of information, and I strongly encourage people to do so. Additionally, this is probably not the most efficient way to achieve this.

Right, on with the show

Here’s the string that I’ll be parsing, and a little code stolen directly from the link at the bottom to show what it looks like:

static void Main(string[] args)
{
    string json = "{\"documents\":[{\"keyPhrases\":[\"Test new bug\"],\"id\":\"1\"}],\"errors\":[]}";
    JsonTextReader reader = new JsonTextReader(new StringReader(json));
    while (reader.Read())
    {
        if (reader.Value != null)
        {
            Console.WriteLine("Token: {0}, Value: {1}", reader.TokenType, reader.Value);
        }
        else
        {
            Console.WriteLine("Token: {0}", reader.TokenType);
        }
    }
 
    Console.ReadLine();
}

The output for this looks like:

Using this, it’s easier to create a routine to manually parse this. Each object can be tracked by using the Start and EndObject tags. Here’s my unit test to check this works:

[TestMethod]
public void TestJSONParse()
{
    // Arrange
    string json = "{\"documents\":[{\"keyPhrases\":[\"Test new bug\"],\"id\":\"1\"}],\"errors\":[]}";
    // Act
    var result = JsonHelper.ParseResponse(json);
 
    // Assert
    Assert.AreEqual(1, result.Count());
    Assert.AreEqual(1, result.Keys.First());
    string expectedPhrase = result.Values.First().First().ToString();
    Assert.AreEqual("Test new bug", expectedPhrase, false);
}

And here’s the code itself:

/// <summary>
/// Parse the following JSON
/// {"documents":[{"keyPhrases":["Test new bug"],"id":"1"}],"errors":[]}
/// </summary>
/// <param name="response"></param>
/// <returns></returns>
public static Dictionary<int, List<string>> ParseResponse(string response)
{
    Dictionary<int, List<string>> dict = new Dictionary<int, List<string>>();
    object readerValue;
 
    if (!string.IsNullOrWhiteSpace(response))
    {
        JsonTextReader reader = new JsonTextReader(new StringReader(response));
        int? currentValue = null;
        List<string> currentList = null;
 
        while (reader.Read())
        {
            readerValue = reader.Value;
 
            switch (reader.TokenType)
            {
                case JsonToken.PropertyName:                            
                    if (readerValue.ToString() == "id")
                    {
                        reader.Read();
                        currentValue = int.Parse(reader.Value.ToString());                                
                    }
                    else if (readerValue.ToString() == "keyPhrases")
                    {
                        // Do nothing
                    }
                    else if (readerValue.ToString() == "errors")
                    {
                        currentValue = null;
                    }
                    break;
 
                case JsonToken.String:                            
                    currentList.Add(reader.Value.ToString());
                    break;
 
                case JsonToken.StartArray:
                    currentList = new List<string>();
                    break;
 
                case JsonToken.StartObject:
                    currentList = null;
                    currentValue = null;
                    break;
 
                case JsonToken.EndObject:
                    if (currentValue.HasValue)
                    {
                        dict.Add(currentValue.Value, currentList);
                    }
                    break;
            }
        }
    }
 
    return dict;
}

It is messy, and it is error prone, and it would be better done by creating classes and serialising it; however, I’d never attempted to do this manually before, and it’s generally nice to do things the hard way, that way, you can appreciate what you get from these tools.

References

http://www.newtonsoft.com/json/help/html/ReadJsonWithJsonTextReader.htm

MVVMCross – Overriding The Default Plugins

Following on from this post I discovered that I did, indeed, have a recursive reference in my game file. After a little searching, I found this excellent article on how to prevent this error, and to make the Json serializer behave rationally.

In MVVMCross, the code to serialize JSON is based on a standard plugin; it looks like this (or at least this is how I am saving my game):

public void Save()
{
    var jsonConv = Mvx.Resolve<IMvxJsonConverter>();            
    string text = jsonConv.SerializeObject(this);
    FileHelper.SaveGameFile(text);
}

The type is registered in App.cs like this:

Mvx.RegisterType<IMvxJsonConverter, MvxJsonConverter>();

What’s the fix?

The above article says the fix is this:

Settings = new JsonSerializerSettings
{
    PreserveReferencesHandling = PreserveReferencesHandling.Objects

(it does go into more detail and I encourage everyone to read it).

So this is a change to MVVM Cross

Might be.

However, you can always use your own Serializer. One of the things that I’ve come to really like about MVVM Cross if that if you don’t like something, just write your own and override that specific thing. I stole this code, verbatim, from MVVM Cross, with the single exception of the one line above:

    public class MyJSonConverter : IMvxJsonConverter
    {
        private static readonly JsonSerializerSettings Settings;

        static KMJSonConverter()
        {
            Settings = new JsonSerializerSettings
            {
                ReferenceLoopHandling = Newtonsoft.Json.ReferenceLoopHandling.Serialize,
                Converters = new List<JsonConverter>
                        {
                            new MvxEnumJsonConverter(),
                        },
                DateFormatHandling = DateFormatHandling.IsoDateFormat,
                PreserveReferencesHandling = PreserveReferencesHandling.Objects
            };
        }

        public T DeserializeObject<T>(string inputText)
        {
            return JsonConvert.DeserializeObject<T>(inputText, Settings);
        }

        public string SerializeObject(object toSerialise)
        {
            return JsonConvert.SerializeObject(toSerialise, Formatting.None, Settings);
        }

        public object DeserializeObject(Type type, string inputText)
        {
            return JsonConvert.DeserializeObject(inputText, type, Settings);
        }
    }

Now, in App.cs, just change how it is registered:

Mvx.RegisterType<IMvxJsonConverter, MyJSonConverter>();
//Mvx.RegisterType<IMvxJsonConverter, MvxJsonConverter>();

Job done. It now works!

Detecting Recursion – JSON.Net Stack Overflow

Trying to save the game class of a game that I’m working on, I got a Stack Overflow error. I’m using MVVMCross, and my code looked like this:

        public void Save()
        {
            var jsonConv = Mvx.Resolve<IMvxJsonConverter>();
            string text = jsonConv.SerializeObject(this);
            FileHelper.SaveGameFile(text);
        }

The problem was that I got this error:

An unhandled exception of type ‘System.StackOverflowException’ occurred in mscorlib.dll

stackoverflow

I had a pretty good idea why. My game features a large population of “people”. Some of these people relate to each other; for example, they are parents, children, employers, etc. My guess was that I’d somehow messed up the creation routine and ended up with a recursive reference. (As it happened, far from messing it up, I hadn’t considered that a spouse relationship is recursive by definition!)

The Problem

The problem was that I had a starting population of 10,000 people. There are other ways to solve this: representing the reference between the classes as some kind of index, debugging the creation code, etc… However, I wanted to see if I could write a program to detect this.

My test program looks like this:

    class MyData
    {
        public MyData recursiveRef { get; set; }
        public string test { get;set; }
    }

    class Program
    {
        static void Main(string[] args)
        {
            List<MyData> data = new List<MyData>()
            {
                new MyData() {recursiveRef = null, test="non-recursive" },
                new MyData() {recursiveRef = null, test="recursive ref 1" },
                new MyData() {recursiveRef = null, test="recursive ref 2" },
                new MyData() {recursiveRef = null, test="recursive ref 3" },
                new MyData() {recursiveRef = null, test="recursive ref back to 1" }
            };
            data[1].recursiveRef = data[2];
            data[2].recursiveRef = data[3];
            data[3].recursiveRef = data[4];
            data[4].recursiveRef = data[1];

            SerialiseData(data);

            Console.ReadLine();
        }

        private static void SerialiseData(List<MyData> data)
        {
            JsonSerializerSettings settings = new JsonSerializerSettings()
            {
                ReferenceLoopHandling = Newtonsoft.Json.ReferenceLoopHandling.Serialize,
                DateFormatHandling = DateFormatHandling.IsoDateFormat,
            };

            var ser = JsonConvert.SerializeObject(data, Formatting.None, settings);
            Console.Write(ser);
        }

Quickly running this will error with a stack overflow (to clarify, slowly running it will result in the same error!).

Detection Routine

Here’s the functions that I wrote to detect recursion:

        /// Itterate the list and call method CheckElement to analyse each element
        private static bool DetectRecursion<T>(IEnumerable<T> data)
        {
            Type typeT = typeof(T);

            foreach (T element in data)
            {
                if (CheckElement<T>(typeT, element, typeT, element))
                {
                    Console.WriteLine("Recursion found - exiting");
                    Console.ReadLine();
                    return true;
                }
            }

            return false;
        }

        private static List<object> recursionChain;

        /// Method recursively traverses the object to determine if there are any recursove references
        private static bool CheckElement<T>(Type baseType, T baseElement, Type checkType, object checkElement)
        {
            PropertyInfo[] piArr = checkType.GetProperties();
            foreach (PropertyInfo pi in piArr)
            {
                Console.WriteLine("Checking {0}, type {1}", pi.Name, pi.PropertyType);
                if (pi.PropertyType != baseType)
                    continue;

                var val = pi.GetValue(checkElement);
                if (val == null) continue;

                Console.WriteLine("Value for {0} is {1}", pi.Name, val.ToString());

                Type piType = val.GetType();

                if (piType == baseType && val.Equals(baseElement))
                {
                    return true;
                }

                if (CheckRecursionChain(val, piType)) return true;

                if (CheckElement<T>(baseType, baseElement, piType, val))
                {
                    Console.WriteLine("Successfully found recursive element {0}, {1}", piType.ToString(), val.ToString());
                    return true;
                }
            }

            return false;
        }

        /// Check the static recursion chain for a match
        private static bool CheckRecursionChain(object val, Type piType)
        {            
            if (Program.recursionChain != null && Program.recursionChain.Contains(val))
            {
                Console.WriteLine("Successfully found recursive element {0}, {1}", piType.ToString(), val.ToString());
                return true;
            }

            if (Program.recursionChain == null)
                Program.recursionChain = new List<object>();

            Program.recursionChain.Add(val);
            
            return false;
        }

This is, admittedly, not a simple or short piece of code; having said that, it doesn’t do anything complex, once you’re happy with the reflection, the rest is just a recursive tree.

To use this, simply call `DetectRecursion` in the above test program before calling serialize (or instead of).

...
DetectRecursion(data);

//SerialiseData(data);

...

MVVM Cross Basics – Passing Complex Parameters During Navigation

Having covered basic navigation, I found that I needed to pass my game state from view-model to view-model. I initially thought that I could simply do this as follows:

    private void CallVM2()
    {
        MyObj newObj = new MyObj();
        IMyService myService = new MyService();

		Dictionary<string, object> p = new Dictionary<string, object>()
        {
            {"MyObj", newObj},
            {"MyService", myService}
        };
		ShowViewModel<ViewModel2>(p);
    }

The code for ViewModel2:

    public void Init(Dictionary<string, object> p)
    {
    }

Anyway, it turns out you can’t. Init does get fired, but if you pass anything more complex than a string, it just bins the parameter.

The answer appears to be to create a class and to serialise it using JSON.NET:

http://stackoverflow.com/questions/19058173/passing-complex-navigation-parameters-with-mvvmcross-showviewmodel

After implementing this suggestion, I ended up with something like this:

    public abstract class BaseViewModel : MvxViewModel
    {
        private const string ParameterName = "parameter";

        public BaseViewModel()
        {
        }

        protected void ShowViewModel<TViewModel>(object parameter) where TViewModel : IMvxViewModel
        {
            var text = Mvx.Resolve<IMvxJsonConverter>().SerializeObject(parameter);
            base.ShowViewModel<TViewModel>(text);
        }

        public void Init(string parameter)
        {
            if (parameter == null || parameter.Length == 0) return;

            IMvxJsonConverter converter = Mvx.Resolve<IMvxJsonConverter>();
            NavigationParameter deserialized = converter.DeserializeObject<NavigationParameter>(parameter);
            RealInit(deserialized);
            
        }
    }

If you don’t register the converter in the IoC container (as I initially didn’t), then you’ll get this error:

A first chance exception of type ‘Cirrious.CrossCore.Exceptions.MvxIoCResolveException’ occurred in Cirrious.CrossCore.DLL

Additional information: Failed to resolve type Cirrious.CrossCore.Platform.IMvxJsonConverter

The fix:

    public abstract class BaseViewModel : MvxViewModel
    {
        private const string ParameterName = "parameter";

        public BaseViewModel()
        {
            Mvx.RegisterType<IMvxJsonConverter, MvxJsonConverter>();
        }

Now, if you override the `RealInit` method, you’ll see that the parameters are, indeed, available. However, I had a further problem; this time being with JSON.NET. As you can see from the above structure, it’s a dictionary of objects. So, when running this, it will certainly pass through the object, but it’ll just be the serialised string.

And…?

I’m pretty sure that I could get around this using a combination of generics and a custom Javascript serialisation library, but I’ve got a game to write, so I’m going to stick with declaring the Navigation parameter as follows:

   public class NavigationParameter
    {
        public MyObject ObjectParameter { get; set; }
        public MyService ServiceParameter { get; set; } 
    }

It’s worth bearing in mind that you can’t use interfaces either in the parameters; JSON.NET needs a concrete class.