I was looking through Fleece and a few other similar JSON-parsing libraries, and it seems like they’re pretty popular in the F# community (and I’ve used something like that professionally before). In my current code base though, we just use Json.NET as it makes serializing and deserializing a one-line affair. Besides functional purity, is there a compelling case for these kinds of manual mapping libraries? To me it seems like the boilerplate and possibility of spelling errors makes it not worth it, but I’m wondering if there’s something I’ve missed.
Hello,
small disclaimer first, I am the creator of Thoth.Json.
I will give you the same explanation I give when people ask me why I created Thoth.Json. So it will be mainly focused on this library.
First, I created Thoth.Json because I never liked auto-magic serialization/deserialization 
Auto-magic serialization/deserialization:
- Force you to have a 1 to 1 match between your JSON structure and your types. So if the JSON structure makes no sense for your application, you will need to deserialize it first and then map it to your types.
- In general, it doesn’t check if the JSON has a valid value for the expected types.
- Overriding the default serialization format is not easy.
- Errors aren’t really useful and make it kind of hard to know where it happens and why.
- In general, people never
try ... catch...JSON parsing error. Because they assume it will always work.
Example of error reported by Thoth.Json:
Error at: `$.user.firstname`
Expecting an object with path `user.firstname` but instead got:
{
"user": {
"name": "maxime",
"age": 25
}
}
Node `firstname` is unkown.
You know where is the error exactly and why it’s failing. You can even paste the error path in http://jsonpath.com/ to see which item is failing. This is pretty useful when you have a list of X items.
I understand why people like auto-magic serialization/deserialization and Thoth.Json supports it because it makes sense when you have F# on both the client and the server. But if you are not owning or controlling the API you receive then you should always validate the received data.
You got also full control over how you use the JSON. For example, you check the value of the field version and then call decodeVersion1, decodeVersion2, etc.
2 Likes
While Json.NET does provide a lot of simplicity (and I use it in many cases), there are a few reasons that I reach for libraries like Chiron, Fleece, and Thoth.
Having to make DTO’s for serialization is fairly simple at the onset. But when API’s need to change, you end up with DTO’s for all the different versions you want to handle, and it can become an explosion of types to manage, along with the baggage that comes with them. Lots of little mappers have to be written anyway, only they become haphazardly mixed in with the automatic serialization to the point you have to dig through lots of classes to find out how certain fields are even deserialized.
Besides versioning, there is error handling. Getting a very general serialization exception is usually not very helpful to API consumers, but it’s basically all you get. When you are building your mappings in code, you can put your error handling right there as well, and raise useful errors.
Then there is the type safety. You can often get nulls and you’ll need to deal with those at the serialization boundary. That means it’s subjective how much you, as a developer, decide to validate the DTO’s. These functional mappers generally won’t even allow nulls - you’ll have to give them an option field instead, and then it’s no longer up to you as a developer to remember to check because the compiler will enforce them.
There are also a lot of API’s out there that return highly variant types. As an example, take the Hacker News API: https://github.com/HackerNews/API. The items returned in JSON can be of all sorts, which would mean a lot of converters to manage in JSON.NET, all hinged on a type descriptor. To me, it’s a lot clearer to do this in a match expression: https://github.com/ninjarobot/suave-chiron-netcore/blob/master/SuaveChironNetcore/HNProvider.fs#L79
There is certainly personal preference, whether you want to manage a lot of classes or not, and how much you can leverage the compiler to enforce type safety, and there are cases where you don’t need all of those and an automapper can be a good solution.
1 Like
I use manual serialization for:
- Better space-efficiency. Json.Net is not efficient for F# code and even more efficient approaches like FSharpLu have verbose Json, with for example a lot of maps when there could be arrays.
- Better performance. Reflection is slow.
- Safer versioning. If a type changes you can adjust the old deserializer and add a new one.
I don’t use Json because it’s slower and more space-inefficient than binary serilization (CBOR). I also don’t use helper libraries as working with a Json or CBOR object directly is easy.
1 Like
Thanks @dave.curylo, @MangelMaxime, and @charlesroddie for the comprehensive answers.
It seems that a significant amount of guidance in the F# community recommends using DTOs. See https://fsharpforfunandprofit.com/posts/serializating-your-domain-model/ and https://eiriktsarpalis.wordpress.com/2018/10/30/a-contract-pattern-for-schemaless-datastores/ . Would you say that using this fromDomain/toDomain pattern with DTOs achieves mostly the same thing as these manual serializers?
This is the first time I see a DTOs scheme, and I think using Thoth.Json at work we are skipping the DTOs -> Domain phase.
At work, it’s directly JSON -> Domain because I can access the part I want from the JSON directly.
For example, I can say, go to $.data.resource1.subResource without needing to create type for data or resource1, etc. I can directly access the data by providing the path.
Would you say that using this fromDomain/toDomain pattern with DTOs achieves mostly the same thing as these manual serializers?
The DTO that I have seen are inefficient at handling DU data, with fields for all cases which take null values for the inactive cases.
DTOs may be best for achieving an object-relational mapping as their tabular structure fits. If you serialize them without additional thought you will get maps representing tables which is very inefficient.
In general DTOs have more type safety than manual serialization, but lower flexibility, efficiency, and performance.
In general DTOs have more type safety than manual serialization.
One distinctive Fleece feature is that it allows you to compose codecs in a functional way, using standard combinators.
These codecs use a single method for both serializing and deserializing, in contrast to the pairs toJson (or toDomain) / ofJson (ofDomain).
By combining codecs this way, you write less code but most importantly you get guarantees that serialization will be always in sync with deserialization.
A consequence of this is that there’s no need to create and maintain tedious roundtrip tests in your project.
1 Like
I see 3 main types of JSON parsers, each fitting different use cases.
- Auto-map to CLR types (records or classes), like Newtonsoft
- Manually-specified parsers, like others mentioned
- Dynamic access, such as JsonValue
Auto-mappers are nice to use with DTO types. I use DTOs as wire-format (untrusted, unvalidated) representations of incoming messages. For internal/private APIs where the same team controls both front and back ends, it is easy/common to keep the sides in lock step. Even when I use a different language for the front-end, I just remake the type in that language and auto-map it there also. If I used parsers instead, I would essentially have to maintain the exact same structural information twice per language (once in the DTO, once in parser). So it is a strict savings to use an auto mapper for me in this type of setup. Re: reflection is slow. Most auto-mappers cache the type data so the reflection cost is only paid once.
Manual parsers are nice to use for “fuzzy” data types. Many public APIs (for example, social providers) have wildly varying data structures that don’t cleanly map to a single DTO, so parsers are a great fit. Also if you expose a public API, it could make sense to use manual mappers for evolvability of your API. (To counterpoint, I currently lean towards Rich Hickey’s ideas about versioning, but I will reserve judgement for when I actually expose a public API. )
And lastly, there are cases where you want to hone in on specific pieces of the JSON data without doing a full parse. For example, you want to examine a specific property for filtering / routing purposes. Or you want to pre-process some data from the request without losing the original payload, so you can pass it on as-is. That would be more of a case for dynamically accessing the JSON data such as what is done with JsonValue.
1 Like