MESSAGEPACK * Try! * Spec * #msgpack IT'S LIKE JSON. BUT FAST AND SMALL. MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON. But it's faster and smaller. Small integers are encoded into a single byte, and typical short strings require only one extra byte in addition to the strings
themselves. NEXT: MessagePack is supported by over 50 programming languages and
environments. See list of implementations. Redis scripting has support for MessagePack because it is a fast and compact serialization format with a simple to implement specification. I liked it so much that I implemented a MessagePack C extension for Lua just to include it into Redis. Salvatore Sanfilippo, creator of Redis Fluentd uses MessagePack for all internal data representation. It's crazy fast because of zero-copy optimization of msgpack-ruby. Now MessagePack is an essential component of Fluentd to achieve high performance and flexibility at the same time. Sadayuki Furuhashi, creator of Fluentd Treasure Data built a multi-tenant database optimized for analytical queries using MessagePack. The schemaless database is growing by billions of records every month. We also use MessagePack as a glue between components. Actually we just wanted a fast replacement of JSON, and MessagePack is simply useful.
Kazuki Ohta, CTO MessagePack has been simply invaluable to us. We use MessagePack + Memcache to cache many of our feeds on Pinterest. These feeds are compressed and very quick to unpack thanks to MessagePack while Memcache gives us fast atomic pushes. Marty Weiner, Software Engineer Also use MessagePack? Waiting for your testimonial! Tweets about "msgpack OR MessagePack"
LANGUAGES * Java msgpack * D msgpack * Python msgpack * Erlang msgpack * Ruby msgpack * Scala msgpack * Haskell msgpack * Ruby/C++ mneumann * Haxe aaulia * C# msgpack * C/C++ msgpack * OCaml msgpack * Smalltalk msgpack * ActionScript3 loteixeira
* PHP msgpack * Go vmihailenco * Rust mneumann * Elixir mururu * Scheme ktakashi * C++11 Lichtso * Go ugorji * mruby suzukaze * Elixir lexmag * Rails jingweno * Python vsergeev * Lua markstinson * Julia JuliaIO * Clojure edma2 * SML tkob * C camgunz * Dart danellis * HHVM reeze * F# Gab-km * Jackson-dataformat komamitsu
* Swift briandw * Objective-C gabriel
* Node mcollina * Haskell rodrigosetti
* Pascal ymofen * Delphi chinawsb * Qt romixlab * Go tinylib * Shell jakm * C# ymofen * Elixir ymofen * Scala msgpack4z * scala-native msgpack4z * Python/Twisted jakm
* J jonghough * Swift a2 * Nim akiradeveloper
* C ludocode * Rust 3Hren * Nim jangko * Crystal crystal-community
* mruby Asmod4n * JavaScript kawanet * UNIX Shell ludocode
* PHP rybakit * Scala.js msgpack4z * Swift michael-yuji
* C# caesay * C++11 jonathonl * Arduino C HEADS-project * Python polyglotted * ActionScript3 st3v0
* R eddelbuettel * Crystal steakknife
* C++11 ar90n * Perl 6 pierre-vigier * msgpack-inspect tagomoris * Swift tris-foundation
* C clwi * C# neuecc * F# pocketberserker
* D jpf91 * Node MobiltronInc * R crowding * Retrofit converter (Java) komamitsu * PostgreSQL patriksimek
* Swift-4 Dev1an * Racket HiPhish * R skyformat99 * Pony SeanTAllen * Objective-C forzafootball
* Python lekma * Go dchenk * Swift, java compatibel cherrywoods
* Matlab bastibe * Kotlin davethomas11
* LabVIEW mradziwo * ClojureScript pkcsecurity * SION,Swift dankogai
* JavaScript ygoe * Go shamaton * Lua kieselsteini * GNU Guile HiPhish * Javascript/NodeJS davalapar
* Dart knopp * Go ugorji * Swift hirotakan API JANGKO/MSGPACK4NIM HTTPS://GITHUB.COM/JANGKO/MSGPACK4NIM
MSGPACK4NIM MessagePack implementation written in pure nim why another implementation? I am fully aware of another msgpack implementation written in nim. But I want something easier to use. Another motivation come from the nim language itself. The current version of nim compiler offer many improvements, including 'generics ' specialization. I found out nim compiler is smart enough to make serialization/deserialization to/from msgpack easy and convenient. REQUIREMENT: nim ver 0.11.2 or later
EXAMPLE import msgpack4nim, streams
type #lets try with a rather complex object CustomType = object
count: int content: seq name: string ratio: float attr: array ok: bool proc initCustomType(): CustomType =
result.count = -1 result.content = @ result.name = "custom" result.ratio = 1.0 for i in 0..5: result.attr = i
result.ok = false var x = initCustomType() #you can use another stream compatible #class here e.g. FileStream var s = newStringStream() s.pack(x) #here the magic happened
s.setPosition(0) var xx: CustomType s.unpack(xx) #and here too
assert xx == x echo "OK ", xx.name see? you only need to call 'pack' and 'unpack', and the compiler do the hard work for you. Very easy, convenient, and works well if you think setting up a StringStream too much for you, you can simply call pack(yourobject) and it will return a string containing
msgpack data. var a = @ var buf = pack(a)
var aa: seq unpack(buf, aa) assert a == aa in case the compiler cannot decide how to serialize or deserialize your very very complex object, you can help it in easy way by defining your own handler pack_type/unpack_type
type #not really complex, just for example mycomplexobject = object
a: someSimpleType b: someSimpleType #help the compiler to decide proc pack_type*(s: Stream, x: mycomplexobject) = s.pack(x.a) # let the compiler decide s.pack(x.b) # let the compiler decide #help the compiler to decide proc unpack_type*(s: Stream, x: var complexobject) =
s.unpack(x.a) s.unpack(x.b) var s: newStringStream() var x: mycomplexobject s.pack(x) #pack as usual
s.setPosition(0) s.unpack(x) #unpack as usual
DATA CONVERSION NIM MSGPACK int8/16/32/64 int8/16/32/64 uint8/16/32/64 uint8/16/32/64 true/false/nil true/false/nil procedural type ignored cstring ignored pointer ignored ptr see ref-types ref see ref-types circular ref see ref-types distinct types converted to base type
float32/64 float32/64 string string8/16/32 array/seq array set array range/subrange int8/16/32/64 enum int8/16/32/64 IntSet,Doubly/SinglyLinkedList
array Doubly/SinglyLinkedRing
array Queue,HashSet,OrderedSet
array Table,TableRef map OrderedTable,OrderedTableRef
map StringTableRef map CritBitTree map CritBitTree array object/tuple array/map OBJECT AND TUPLE object and tuple by default converted to msgpack array, however you can tell the compiler to convert it to map by supplying --define:msgpack_obj_to_map nim c --define:msgpack_obj_to_map yourfile.nim or --define:msgpack_obj_to_stream to convert object/tuple fields _value_ into stream of msgpack objects nim c --define:msgpack_obj_to_stream yourfile.nim What this means? It means by default, each object/tuple will be converted to one msgpack array contains field(s) value only without their field(s) name. If you specify that the object/tuple will be converted to msgpack map, then each object/tuple will be converted to one msgpack map contains key-value pairs. The key will be field name, and the value will be
field value. If you specify that the object/tuple will be converted to msgpack stream, then each object/tuple will be converted into one or more msgpack's type for each object's field and then the resulted stream will be concatenated to the msgpack stream buffer. Which one should I use? Usually, other msgpack libraries out there convert object/tuple/record/struct or whatever structured data supported by the language into msgpack array, but always make sure to consult the documentation first. If both of the serializer and deserializer agreed to one convention, then usually there will be no problem. No matter which library/language you use, you can exchange msgpack data among
them. REF-TYPES: _ref something_ : * if ref value is nil, it will be packed into msgpack nil, and when unpacked, you will get nil too * if ref value not nil, it will be dereferenced e.g. pack(val) or
unpack(val) * ref subject to some restriction. see RESTRICTION below * ptr will be treated like ref during pack * unpacking ptr will invoke alloc, so you must dealloc it _circular reference_: altough detecting circular reference is not too difficult(using set of pointers), the current implementation does not provide circular reference detection. If you pack something contains circular reference, you know something bad will happened RESTRICTION: For objects their type is NOT serialized. This means essentially that it does not work if the object has some other runtime type than its compiletime type: import streams, msgpack4nim
type TA = object of RootObj TB = object of TA
f: int var a: ref TA b: ref TB new(b) a = b echo stringify(pack(a)) #produces "" or "{ }" #not "" or '{ "f" : 0 }'
LIMITATION: these types will be ignored:
* procedural type * cstring(it is not safe to assume it always terminated by null)
* pointer these types cannot be automatically pack/unpacked: * _void_ (will cause compile time error) however, you can provide your own handler for cstring and pointer GOTCHAS: because data conversion did not preserve original data types, the following code is perfectly valid and will raise no exception import msgpack4nim, streams, tables, sets, strtabs
type Horse = object legs: int foals: seq attr: Table Cat = object legs: uint8 kittens: HashSet traits: StringTableRef proc initHorse(): Horse =
result.legs = 4 result.foals = @ result.attr = initTable() result.attr = "black" result.attr = "120mph" var stallion = initHorse()
var tom: Cat var buf = pack(stallion) #pack a Horse here
unpack(buf, tom) #abracadabra, it will unpack into a Cat echo "legs: ", $tom.legs echo "kittens: ", $tom.kittens echo "traits: ", $tom.traits
another gotcha: type KAB = object of RootObj
aaa: int bbb: int KCD = object of KAB
ccc: int ddd: int KEF = object of KCD
eee: int fff: int var kk = KEF() echo stringify(pack(kk)) # will produce "{ "eee" : 0, "fff" : 0, "ccc" : 0, "ddd" : 0, "aaa" : 0, "bbb" : 0 }" # not "{ "aaa" : 0, "bbb" : 0, "ccc" : 0, "ddd" : 0, "eee" : 0, "fff" : 0 }"
BIN AND EXT FORMAT this implementation provide function to encode/decode msgpack bin/ext format header, but for the body, you must write it yourself to the
StringStream * proc pack_bin*(s: Stream, len: int) * proc pack_ext*(s: Stream, len: int, exttype: int8) * proc unpack_bin*(s: Stream): int * proc unpack_ext*(s: Stream): tuple import streams, msgpack4nim const exttype0 = 0 var s = newStringStream() var body = "this is the body" s.pack_ext(body.len, exttype0)
s.write(body) #the same goes to bin format s.pack_bin(body.len)
s.write(body) s.setPosition(0) #unpack_ext return tuple let (extype, extlen) = s.unpack_ext() var extbody = s.readStr(extlen) assert extbody == body let binlen = s.unpack_bin() var binbody = s.readStr(binlen) assert binbody == body
STRINGIFY you can convert msgpack data to readable string using stringify
function type Horse = object legs: int speed: int color: string name: string var cc = Horse(legs:4, speed:150, color:"black", name:"stallion") var zz = pack(cc)
echo stringify(zz) the result will be: default: msgpack_obj_to_map defined: { "legs" : 4, "speed" : 150, "color" : "black", "name" : "stallion" } msgpack_obj_to_stream defined: 4 150 "black" "stallion"
TOANY TOANY takes a string of msgpack data or a stream, then it will produce MSGANY which you can interrogate of it's type and value during runtime by accessing it's member MSGTYPE TOANY recognize all valid msgpack message and translate it into a
group of types: msgMap, msgArray, msgString, msgBool, msgBin, msgExt, msgFloat32, msgFloat64, msgInt, msgUint, msgNull for example, MSG is a _msgpack_ data with content , you can interrogate it like this: var a = msg.toAny() assert a.msgType == msgArray assert a.arrayVal.msgType == msgInt assert a.arrayVal.intVal == 1 assert a.arrayVal.msgType == msgString assert a.arrayVal.stringVal == "hello" assert a.arrayVal.msgType == msgMap assert a.arrayVal.mapVal.key.msgType == msgString assert a.arrayVal.mapVal.key.stringVal == "a" assert a.arrayVal.mapVal.val.msgType == msgString assert a.arrayVal.mapVal.val.stringVal == "b" enjoy it, happy nim-ing List your implementation here! MSGPACK/MSGPACK-JAVA HTTPS://GITHUB.COM/MSGPACK/MSGPACK-JAVA MESSAGEPACK FOR JAVA QuickStart for msgpack-java is available here
. HOW TO INSTALL You can install msgpack via maven:
... org.msgpack msgpack ${msgpack.version} ... SIMPLE SERIALIZATION/DESERIALIZATION/DUCK TYPING USING VALUE // Create serialize objects. List
src = new ArrayList(); src.add("msgpack"); src.add("kumofs"); src.add("viver"); MessagePack msgpack = new MessagePack(); // Serialize byte raw = msgpack.write(src); // Deserialize directly using a template List dst1 = msgpack.read(raw, Templates.tList(Templates.TString)); System.out.println(dst1.get(0)); System.out.println(dst1.get(1)); System.out.println(dst1.get(2)); // Or, Deserialze to Value then convert type. Value dynamic = msgpack.read(raw); List dst2 = new Converter(dynamic) .read(Templates.tList(Templates.TString)); System.out.println(dst2.get(0)); System.out.println(dst2.get(1)); System.out.println(dst2.get(2)); MSGPACK/MSGPACK-D HTTPS://GITHUB.COM/MSGPACK/MSGPACK-D MESSAGEPACK FOR D MessagePack is a binary-based JSON-like serialization library. MessagePack for D is a pure D implementation of MessagePack. FEATURES * Small size and High performance * Zero copy serialization / deserialization * Streaming deserializer for non-contiguous IO situation * Supports D features (Ranges, Tuples, real type) Note: The real type is only supported in D. Don't use the real type when communicating with other programming languages. Note that Unpacker will raise an exception if a loss of precision occurs. CURRENT LIMITATIONS * No circular references support * If you want to use the LDC compiler, you need at least version 0.15.2 beta2 INSTALL Use dub to add it as a dependency: % dub install msgpack-d USAGE Example code can be found in the example directory. The documentation can be found here PACK / UNPACK msgpack-d is very simple to use. Use pack for serialization, and unpack for deserialization: import std.file; import msgpack; struct S { int x; float y; string z; } void main() { S input = S(10, 25.5, "message"); // serialize data ubyte inData = pack(input); // write data to a file write("file.dat", inData); // read data from a file ubyte outData = cast(ubyte)read("file.dat"); // unserialize the data S target = outData.unpack!S(); // verify data is the same assert(target.x == input.x); assert(target.y == input.y); assert(target.z == input.z); } Feature: Skip serialization/deserialization of a specific field. Use the @nonPacked attribute: struct User { string name; @nonPacked int level; // pack / unpack will ignore the 'level' field } Feature: Use your own serialization/deserialization routines for custom class and struct types. msgpack-d provides the functions registerPackHandler / registerUnpackHandler to allow you to use custom routines during the serialization or deserialization of user-defined class and struct types. This feature is especially useful when serializing a derived class object when that object is statically typed as a base class object. For example: class Document { } class XmlDocument : Document { this() { } this(string name) { this.name = name; } string name; } void xmlPackHandler(ref Packer p, ref XmlDocument xml) { p.pack(xml.name); } void xmlUnpackHandler(ref Unpacker u, ref XmlDocument xml) { u.unpack(xml.name); } void main() { /// Register the 'xmlPackHandler' and 'xmlUnpackHandler' routines for /// XmlDocument object instances. registerPackHandler!(XmlDocument, xmlPackHandler); registerUnpackHandler!(XmlDocument, xmlUnpackHandler); /// Now we can serialize/deserialize XmlDocument object instances via a /// base class reference. Document doc = new XmlDocument("test.xml"); auto data = pack(doc); XmlDocument xml = unpack!XmlDocument(data); assert(xml.name == "test.xml"); // xml.name is "test.xml" } THE PACKERIMPL / UNPACKER / STREAMINGUNPACKER TYPES These types are used by the pack and unpack functions. See the documentation of PackerImpl , Unpacker and StreamingUnpacker for more details. LINKS * The MessagePack Project The official MessagePack protocol website. * msgpack-d's issue tracker Use this issue tracker to review and file bugs in msgpack-d. * MessagePack's Github Other language bindings and implementations of the msgpack protocol can be found here. COPYRIGHT Copyright (c) 2010- Masahiro Nakagawa LICENSE Distributed under the Boost Software License, Version 1.0 . MSGPACK/MSGPACK-PYTHON HTTPS://MSGPACK.ORG/ MESSAGEPACK FOR PYTHON WHAT'S THIS MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON. But it's faster and smaller. This package provides CPython bindings for reading and writing MessagePack data. VERY IMPORTANT NOTES FOR EXISTING USERS PyPI package name TL;DR: When upgrading from msgpack-0.4 or earlier, don't do pip install -U msgpack-python. Do pip uninstall msgpack-python; pip install msgpack instead. Package name on PyPI was changed to msgpack from 0.5. I upload transitional package (msgpack-python 0.5 which depending on msgpack) for smooth transition from msgpack-python to msgpack. Sadly, this doesn't work for upgrade install. After pip install -U msgpack-python, msgpack is removed and import msgpack fail. Deprecating encoding option encoding and unicode_errors options are deprecated. In case of packer, use UTF-8 always. Storing other than UTF-8 is not recommended. For backward compatibility, you can use use_bin_type=False and pack bytes object into msgpack raw type. In case of unpacker, there is new raw option. It is True by default for backward compatibility, but it is changed to False in near future. You can use raw=False instead of encoding='utf-8'. Planned backward incompatible changes When msgpack 1.0, I planning these breaking changes: * packer and unpacker: Remove encoding and unicode_errors option. * packer: Change default of use_bin_type option from False to True. * unpacker: Change default of raw option from True to False. * unpacker: Reduce all max_xxx_len options for typical usage. * unpacker: Remove write_bytes option from all methods. To avoid these breaking changes breaks your application, please: * Don't use deprecated options. * Pass use_bin_type and raw options explicitly. * If your application handle large (>1MB) data, specify max_xxx_len options too. INSTALL $ pip install msgpack PyPy msgpack provides a pure Python implementation. PyPy can use this. Windows When you can't use a binary distribution, you need to install Visual Studio or Windows SDK on Windows. Without extension, using pure Python implementation on CPython runs slowly. For Python 2.7, Microsoft Visual C++ Compiler for Python 2.7 is recommended solution. For Python 3.5, Microsoft Visual Studio 2015 Community Edition or Express Edition can be used to build extension module. HOW TO USE One-shot pack & unpack Use packb for packing and unpackb for unpacking. msgpack provides dumps and loads as an alias for compatibility with json and pickle. pack and dump packs to a file-like object. unpack and load unpacks from a file-like object. >>> import msgpack >>> msgpack.packb(, use_bin_type=True) '\x93\x01\x02\x03' >>> msgpack.unpackb(_, raw=False) unpack unpacks msgpack's array to Python's list, but can also unpack to tuple: >>> msgpack.unpackb(b'\x93\x01\x02\x03', use_list=False, raw=False) (1, 2, 3) You should always specify the use_list keyword argument for backward compatibility. See performance issues relating to use_list option below. Read the docstring for other options. Streaming unpacking Unpacker is a "streaming unpacker". It unpacks multiple objects from one stream (or from bytes provided through its feed method). import msgpack from io import BytesIO buf = BytesIO() for i in range(100): buf.write(msgpack.packb(i, use_bin_type=True)) buf.seek(0) unpacker = msgpack.Unpacker(buf, raw=False) for unpacked in unpacker: print(unpacked) Packing/unpacking of custom data type It is also possible to pack/unpack custom data types. Here is an example for datetime.datetime. import datetime import msgpack useful_dict = { "id": 1, "created": datetime.datetime.now(), } def decode_datetime(obj): if b'__datetime__' in obj: obj = datetime.datetime.strptime(obj, "%Y%m%dT%H:%M:%S.%f") return obj def encode_datetime(obj): if isinstance(obj, datetime.datetime): return {'__datetime__': True, 'as_str': obj.strftime("%Y%m%dT%H:%M:%S.%f")} return obj packed_dict = msgpack.packb(useful_dict, default=encode_datetime, use_bin_type=True) this_dict_again = msgpack.unpackb(packed_dict, object_hook=decode_datetime, raw=False) Unpacker's object_hook callback receives a dict; the object_pairs_hook callback may instead be used to receive a list of key-value pairs. Extended types It is also possible to pack/unpack custom data types using the EXT type. >>> import msgpack >>> import array >>> def default(obj): ... if isinstance(obj, array.array) and obj.typecode == 'd': ... return msgpack.ExtType(42, obj.tostring()) ... raise TypeError("Unknown type: %r" % (obj,)) ... >>> def ext_hook(code, data): ... if code == 42: ... a = array.array('d') ... a.fromstring(data) ... return a ... return ExtType(code, data) ... >>> data = array.array('d', ) >>> packed = msgpack.packb(data, default=default, use_bin_type=True) >>> unpacked = msgpack.unpackb(packed, ext_hook=ext_hook, raw=False) >>> data == unpacked True Advanced unpacking control As an alternative to iteration, Unpacker objects provide unpack, skip, read_array_header and read_map_header methods. The former two read an entire message from the stream, respectively de-serialising and returning the result, or ignoring it. The latter two methods return the number of elements in the upcoming container, so that each element in an array, or key-value pair in a map, can be unpacked or skipped individually. Each of these methods may optionally write the packed data it reads to a callback function: from io import BytesIO def distribute(unpacker, get_worker): nelems = unpacker.read_map_header() for i in range(nelems): # Select a worker for the given key key = unpacker.unpack() worker = get_worker(key) # Send the value as a packed message to worker bytestream = BytesIO() unpacker.skip(bytestream.write) worker.send(bytestream.getvalue()) NOTES string and binary type Early versions of msgpack didn't distinguish string and binary types (like Python 1). The type for representing both string and binary types was named RAW. For backward compatibility reasons, msgpack-python will still default all strings to byte strings, unless you specify the use_bin_type=True option in the packer. If you do so, it will use a non-standard type called BIN to serialize byte arrays, and RAW becomes to mean STR. If you want to distinguish BIN and RAW in the unpacker, specify raw=False. Note that Python 2 defaults to byte-arrays over Unicode strings: >>> import msgpack >>> msgpack.unpackb(msgpack.packb()) >>> msgpack.unpackb(msgpack.packb(, use_bin_type=True), raw=False) This is the same code in Python 3 (same behaviour, but Python 3 has a different default): >>> import msgpack >>> msgpack.unpackb(msgpack.packb()) >>> msgpack.unpackb(msgpack.packb(, use_bin_type=True), raw=False) ext type To use the EXT type, pass msgpack.ExtType object to packer. >>> import msgpack >>> packed = msgpack.packb(msgpack.ExtType(42, b'xyzzy')) >>> msgpack.unpackb(packed) ExtType(code=42, data='xyzzy') You can use it with default and ext_hook. See below. NOTE ABOUT PERFORMANCE GC CPython's GC starts when growing allocated object. This means unpacking may cause useless GC. You can use gc.disable() when unpacking large message. use_list option List is the default sequence type of Python. But tuple is lighter than list. You can use use_list=False while unpacking when performance is important. Python's dict can't use list as key and MessagePack allows array for key of mapping. use_list=False allows unpacking such message. Another way to unpacking such object is using object_pairs_hook. DEVELOPMENT Test MessagePack uses pytest for testing. Run test with following command: > $ make test MSGPACK/MSGPACK-ERLANG HTTPS://GITHUB.COM/MSGPACK/MSGPACK-ERLANG MESSAGEPACK ERLANG PREREQUISITES FOR RUNTIME Erlang/OTP , >= 17.0 Also based on the new msgpack spec 0b8f5a . EDIT REBAR.CONFIG TO USE IN YOUR APPLICATION {deps, }. Or as it is now published at hex.pm , just {deps, }. might work. SIMPLE DESERIALIZATION Ham = msgpack:pack(Spam), {ok, Spam} = msgpack:unpack(Ham). STREAM DESERIALIZATION {Term0, Rest0} = msgpack:unpack_stream(Binary), {Term1, Rest1} = msgpack:unpack_stream(Rest0), ... OPTIONS, FOR PACKING AND UNPACKING {spec, new|old} Both for packing and unpacking. Default is new. Major difference between old and new spec is: * raw family (0xa0~0xbf, 0xda, 0xdb) becomes new str family * 0xd9 is new as str8 * new bin space (0xc4, 0xc5, 0xc6 as bin8, bin16, bin32) * new ext space (0xc7, 0xc8, 0xc9 as ext8, ext16, ext32) * new fixext space (0xd4, 0xd5, 0xd6, 0xd7, 0xd8 as fixext1, fixext2, fixext4, fixext8, fixext16), The default is new spec. Old spec mode does not handle these new types but returns error. To use old spec mode, this option is explicitly added. OldHam = msgpack:pack(Spam, ), {ok, Spam} = msgpack:unpack(OldHam, ). {allow_atom, none|pack} Only in packing. Atoms are packed as binaries. Default value is pack. Otherwise, any term including atoms throws badarg. {known_atoms, } Both in packing and unpacking. In packing, if an atom is in this list a binary is encoded as a binary. In unpacking, msgpacked binaries are decoded as atoms with erlang:binary_to_existing_atom/2 with encoding utf8. Default value is an empty list. Even if allow_atom is none, known atoms are packed. {unpack_str, as_binary|as_list} A switch to choose decoded term style of str type when _unpacking_. Only available at new spec. Default is as_list. mode as_binary as_list -----------+------------+------- bin binary() binary() str binary() string() {validate_string, boolean()} Only in unpacking, UTF-8 validation at unpacking from str type will be enabled. Default value is false. {pack_str, from_binary|from_list|none} A switch to choose packing of string() when packing. Only available at new spec. Default is from_list for symmetry with unpack_str option. mode from_list from_binary none -----------+------------+--------------+----------------- binary() bin str*/bin bin string() str*/array array of int array of int list() array array array But the default option pays the cost of performance for symmetry. If the overhead of UTF-8 validation is unacceptable, choosing none as the option would be the best. * * Tries to pack as str if it is a valid string(). {map_format, map|jiffy|jsx} Both at packing and unpacking. Default value is map. msgpack:pack(#{ <<"key">> => <<"value">> }, ). msgpack:pack(#{ <<"key">> => <<"value">> }, ). msgpack:pack({}, ), msgpack:pack(, ). {ext, {msgpack_ext_packer(), msgpack_ext_unpacker()}|module()} At both. The default behaviour in case of facing ext data at decoding is to ignore them as its length is known. Now msgpack-erlang supports ext type. Now you can serialize everything with your original (de)serializer. That will enable us to handle erlang- native types like pid(), ref() contained in tuple(). See test/msgpack_ext_example_tests.erl for example code. Packer = fun({ref, Ref}, Opt) when is_reference(Ref) -> {ok, {12, term_to_binary(Ref)}} end, Unpacker = fun(12, Bin) -> {ok, {ref, binary_to_term(Bin)}} end, Ref = make_ref(), Opt = , {ok, {ref, Ref}} = msgpack:unpack(msgpack:pack({ref, Ref}, Opt), Opt). MISC Float type The Float type of Message Pack represents IEEE 754 floating point number, so it includes Nan and Infinity. In unpacking, msgpack-erlang returns nan, positive_infinity and negative_infinity. LICENSE Apache License 2.0 RELEASE NOTES 0.7.0 * Support nan, positive_infinity and negative_infinity 0.6.0 * Support OTP 19.0 0.5.0 * Renewed optional arguments to pack/unpack interface. This is incompatible change from 0.4 series. 0.4.0 * Deprecate nil * Moved to rebar3 * Promote default map unpacker as default format when OTP is >= 17 * Added QuickCheck tests * Since this version OTP older than R16B03-1 are no more supported 0.3.5 / 0.3.4 * 0.3 series will be the last versions that supports R16B or older versions of OTP. * OTP 18.0 support * Promote default map unpacker as default format when OTP is >= 18 0.3.3 * Add OTP 17 series to Travis-CI tests * Fix wrong numbering for ext types * Allow packing maps even when {format,map} is not set * Fix Dialyzer invalid contract warning * Proper use of null for jiffy-style encoding/decoding 0.3.2 * set back default style as jiffy * fix bugs around nil/null handling 0.3.0 * supports map new in 17.0 * jiffy-style maps will be deprecated in near future * set default style as map 0.2.8 0.2 series works with OTP 17.0, R16, R15, and with MessagePack's new and old format. But does not support map type introduced in OTP 17.0. It also supports JSX-compatible mode. MSGPACK/MSGPACK-RUBY HTTPS://GITHUB.COM/MSGPACK/MSGPACK-RUBY MESSAGEPACK FOR RUBY require 'msgpack' msg = .to_msgpack #=> "\x93\x01\x02\x03" MessagePack.unpack(msg) #=> INSTALL gem install msgpack USE CASES * Create REST API returing MessagePack using Rails + RABL * Store objects efficiently in memcached or Redis * Upload data in efficient format from mobile devices. See also MessagePack for Objective-C and Java LINKS * Github * API document STREAMING API # serialize a 2-element array pk = MessagePack::Packer.new(io) pk.write_array_header(2).write(e1).write(e2).flush # deserialize objects from an IO u = MessagePack::Unpacker.new(io) u.each { |obj| ... } # event-driven deserialization def on_read(data) @u ||= MessagePack::Unpacker.new @u.feed_each(data) { |obj| ... } end MSGPACK/MSGPACK-SCALA HTTPS://GITHUB.COM/MSGPACK/MSGPACK-SCALA MESSAGEPACK FOR SCALA * Message Pack specification: https://github.com/msgpack/msgpack/blob/master/spec.md QUICK START libraryDependencies += "org.msgpack" %% "msgpack-scala" % "(version)" General usage is the same with msgpack-java. See this example code (Java) . FOR MESSAGEPACK DEVELOPERS Basic sbt commands Enter the sbt console: $ ./sbt Here is a list of sbt commands for daily development: > ~compile # Compile source codes > ~test:compile # Compile both source and test codes > ~test # Run tests upon source code change > ~test-only *MessagePackTest # Run tests in the specified class > ~test-only *MessagePackTest -- -n prim # Run the test tagged as "prim" > project msgpack-scala # Focus on a specific project > package # Create a jar file in the target folder of each project > scalafmt # Reformat source codes > ; coverage; test; coverageReport; coverageAggregate; # Code coverage Publishing > publishLocal # Install to local .ivy2 repository > publish # Publishing a snapshot version to the Sonatype repository > release # Run the release procedure (set a new version, run tests, upload artifacts, then deploy to Sonatype) For publishing to Maven central, msgpack-scala uses sbt-sonatype plugin. Set Sonatype account information (user name and password) in the global sbt settings. To protect your password, never include this file in your project. _$HOME/.SBT/(SBT-VERSION)/SONATYPE.SBT_ credentials += Credentials("Sonatype Nexus Repository Manager", "oss.sonatype.org", "(Sonatype user name)", "(Sonatype password)") MSGPACK/MSGPACK-HASKELL HTTPS://GITHUB.COM/MSGPACK/MSGPACK-HASKELL MESSAGEPACK FOR HASKELL This is an implementation of msgpack for Haskell. It containes: * Serializer/Deserializer * RPC INSTALL Execute following instructions: $ cabal update $ cabal install msgpack $ cabal install msgpack-rpc DOCUMENT There are Haddoc documents on Hackage Database. * http://hackage.haskell.org/package/msgpack * http://hackage.haskell.org/package/msgpack-rpc MNEUMANN/MESSAGEPACK HTTPS://GITHUB.COM/MNEUMANN/MESSAGEPACK MESSAGEPACK An alternative msgpack.org implementation for Ruby and C++ AAULIA/MSGPACK-HAXE HTTPS://GITHUB.COM/AAULIA/MSGPACK-HAXE MSGPACK-HAXE MessagePack (http://msgpack.org ) serialization library for Haxe HOW TO INSTALL: Simply use haxelib git to use this github repo or haxelib install msgpack-haxe to use the one in the haxelib repository. SUPPORTED TYPE: * Null * Bool * Int * Float * Object * Bytes * String * Array * IntMap/StringMap EXAMPLE CODE: package; import org.msgpack.MsgPack; class Example { public static function main() { var i = { a: 1, b: 2, c: "Hello World!" }; var m = MsgPack.encode(i); var o = MsgPack.decode(m); trace(i); trace(m.toHex()); trace(o); } } MSGPACK/MSGPACK-CLI HTTPS://GITHUB.COM/MSGPACK/MSGPACK-CLI MESSAGEPACK FOR CLI WHAT IS IT? This is MessagePack serialization/deserialization for CLI (Common Language Infrastructure) implementations such as .NET Framework, Silverlight, Mono (including Moonlight.) This library can be used from ALL CLS compliant languages such as C#, F#, Visual Basic, Iron Python, Iron Ruby, PowerShell, C++/CLI or so. USAGE You can serialize/deserialize objects as following: * Create serializer via MessagePackSerializer.Get generic method. This method creates dependent types serializers as well. * Invoke serializer as following: ** Pack method with destination Stream and target object for serialization. ** Unpack method with source Stream. // Creates serializer. var serializer = MessagePackSerializer.Get(); // Pack obj to stream. serializer.Pack(stream, obj); // Unpack from stream. var unpackedObject = serializer.Unpack(stream); ' Creates serializer. Dim serializer = MessagePackSerializer.Get(Of T)() ' Pack obj to stream. serializer.Pack(stream, obj) ' Unpack from stream. Dim unpackedObject = serializer.Unpack(stream) FOR PRODUCTION ENVIRONMENT, YOU SHOULD INSTANTIATE OWN SERIALIZATIONCONTEXT AND MANAGE ITS LIFETIME. IT IS GOOD IDEA TO TREAT IT AS SINGLETON BECAUSE SERIALIZATIONCONTEXT IS THREAD-SAFE. FEATURES * Fast and interoperable binary format serialization with simple API. * Generating pre-compiled assembly for rapid start up. * Flexible MessagePackObject which represents MessagePack type system naturally. NOTE: AOT SUPPORT IS LIMITED YET. USE SERIALIZER PRE-GENERATION WITH MPU -S UTILITY OR API. If you do not pre-generated serializers, MsgPack for CLI uses reflection in AOT environments, it is slower and it sometimes causes AOT related error (ExecutionEngineException for runtime JIT compilation). DOCUMENTATION See wiki INSTALLATION * Binary files distributed via the NuGet package MsgPack.Cli . * You can extract binary (DLL) file as following: * Download *.zip file from GitHub Release page . * Extract it. * Under the bin directory, binaries are there! * For mono, you can use net461 or net35 drops as you run with. * For Unity, unity3d drop is suitable. HOW TO BUILD For .NET Framework * Install Visual Studio 2017 (Community edition is OK) and 2015 (for MsgPack.Windows.sln). * You must install .NET Framework 3.5, 4.x, .NET Core, and Xamarin dev tools to build all builds successfully. If you do not want to install options, edit element in *.csproj files to exclude platforms you want to exclude. * Install latest .NET Core SDK. * Run with Visual Studio Developer Command Prompt: msbuild MsgPack.sln /t:Restore msbuild MsgPack.sln Or (for Unity 3D drops): msbuild MsgPack.compats.sln /t:Restore msbuild MsgPack.compats.sln Or (for Windows Runtime/Phone drops and Silverlight 5 drops): msbuild MsgPack.Windows.sln /t:Restore msbuild MsgPack.Windows.sln Or (for Xamarin unit testing, you must have Xamarin Business or upper license and Mac machine on the LAN to build on Windows): msbuild MsgPack.Xamarin.sln /t:Restore msbuild MsgPack.Xamarin.sln Or open one of above solution files in your IDE and run build command in it. For Mono * Install latest Mono and .NET Core SDK. * Now, you can build MsgPack.sln and MsgPack.Xamarin.sln with above instructions and msbuild in latest Mono. Note that xbuild does not work because it does not support latest csproj format. Own Unity 3D Build First of all, there are binary drops on github release page, you should use it to save your time. Because we will not guarantee source code organization compatibilities, we might add/remove non-public types or members, which should break source code build. If you want to import sources, you must include just only described on MsgPack.Unity3D.csproj. If you want to use ".NET 2.0 Subset" settings, you must use just only described on MsgPack.Unity3D.CorLibOnly.csproj file, and define CORLIB_ONLY compiler constants. Xamarin Android testing If you run on Windows, it is recommended to use HXM instead of Hyper-V based emulator. You can disable Hyper-V from priviledged (administrator) powershell as follows: Disable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-Hypervisor If you want to use Hyper-V again (such as for Docker for Windows etc.), you can do it by following in priviledged (administrator) powershell: Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-Hypervisor Xamarin Android Trouble shooting tips * Javac shows compilation error. * Rebuild the test project and try it run again. Xamarin iOS testing You must create provisoning profiles in your MacOS devices. See Xamarin documents about provisining for details. There are bundle IDs of current iOS tests: * org.msgpack.msgpack-cli-xamarin-ios-test * org.msgpack.msgpack-cli-xamarin-ios-test-packer * org.msgpack.msgpack-cli-xamarin-ios-test-unpacker * org.msgpack.msgpack-cli-xamarin-ios-test-unpacking * org.msgpack.msgpack-cli-xamarin-ios-test-timestamp * org.msgpack.msgpack-cli-xamarin-ios-test-arrayserialization * org.msgpack.msgpack-cli-xamarin-ios-test-mapserialization _Note that some reflection based serializer tests failed with AOT related limitation._ Xamarin iOS Trouble shooting tips See Xamarin's official trouble shooting docs first. * An error occurred while running unit test project. * Rebuild the project and rerun it. Or, login your Mac again, ant retry it. * It is hard to read English. * You can read localized Xamarin docs with putting {region}-{lang} as the first component of URL path such as https://developer.xamarin.com/ja-jp/guides/.... SEE ALSO * GitHub Page : http://cli.msgpack.org/ * Wiki (documentation) : https://github.com/msgpack/msgpack-cli/wiki * API Reference : http://cli.msgpack.org/doc/top.html * Issue tracker : https://github.com/msgpack/msgpack-cli/issues * MSBuild reference : http://msdn.microsoft.com/en-us/library/0k6kkbsd.aspx * Mono xbuild reference : http://www.mono-project.com/Microsoft.Build MSGPACK/MSGPACK-C HTTPS://GITHUB.COM/MSGPACK/MSGPACK-C MSGPACK FOR C/C++ Version 3.1.1 It's like JSON but smaller and faster. OVERVIEW MessagePack is an efficient binary serialization format, which lets you exchange data among multiple languages like JSON, except that it's faster and smaller. Small integers are encoded into a single byte and short strings require only one extra byte in addition to the strings themselves. EXAMPLE In C: #include #include int main(void) { /* msgpack::sbuffer is a simple buffer implementation. */ msgpack_sbuffer sbuf; msgpack_sbuffer_init(&sbuf); /* serialize values into the buffer using msgpack_sbuffer_write callback function. */ msgpack_packer pk; msgpack_packer_init(&pk, &sbuf, msgpack_sbuffer_write); msgpack_pack_array(&pk, 3); msgpack_pack_int(&pk, 1); msgpack_pack_true(&pk); msgpack_pack_str(&pk, 7); msgpack_pack_str_body(&pk, "example", 7); /* deserialize the buffer into msgpack_object instance. */ /* deserialized object is valid during the msgpack_zone instance alive. */ msgpack_zone mempool; msgpack_zone_init(&mempool, 2048); msgpack_object deserialized; msgpack_unpack(sbuf.data, sbuf.size, NULL, &mempool, &deserialized); /* print the deserialized object. */ msgpack_object_print(stdout, deserialized); puts(""); msgpack_zone_destroy(&mempool); msgpack_sbuffer_destroy(&sbuf); return 0; } See QUICKSTART-C.md for more details. In C++: #include #include #include #include int main(void) { msgpack::type::tuple src(1, true, "example"); // serialize the object into the buffer. // any classes that implements write(const char*,size_t) can be a buffer. std::stringstream buffer; msgpack::pack(buffer, src); // send the buffer ... buffer.seekg(0); // deserialize the buffer into msgpack::object instance. std::string str(buffer.str()); msgpack::object_handle oh = msgpack::unpack(str.data(), str.size()); // deserialized object is valid during the msgpack::object_handle instance is alive. msgpack::object deserialized = oh.get(); // msgpack::object supports ostream. std::cout << deserialized << std::endl; // convert msgpack::object instance into the original type. // if the type is mismatched, it throws msgpack::type_error exception. msgpack::type::tuple dst; deserialized.convert(dst); // or create the new instance msgpack::type::tuple dst2 = deserialized.as >(); return 0; } See QUICKSTART-CPP.md for more details. USAGE C++ Header Only Library When you use msgpack on C++, you can just add msgpack-c/include to your include path: g++ -I msgpack-c/include your_source_file.cpp If you want to use C version of msgpack, you need to build it. You can also install the C and C++ versions of msgpack. Building and Installing Install from git repository USING THE TERMINAL (CLI) You will need: * gcc >= 4.1.0 * cmake >= 2.8.0 C and C++03: $ git clone https://github.com/msgpack/msgpack-c.git $ cd msgpack-c $ cmake . $ make $ sudo make install If you want to setup C++11 or C++17 version of msgpack instead, execute the following commands: $ git clone https://github.com/msgpack/msgpack-c.git $ cd msgpack-c $ cmake -DMSGPACK_CXX=ON . $ sudo make install MSGPACK_CXX flags are not affected to installing files. Just switching test cases. All files are installed in every settings. When you use the C part of msgpack-c, you need to build and link the library. By default, both static/shared libraries are built. If you want to build only static library, set BUILD_SHARED_LIBS=OFF to cmake. If you want to build only shared library, set `BUILD_SHARED_L GUI on Windows Clone msgpack-c git repository. $ git clone https://github.com/msgpack/msgpack-c.git or using GUI git client. e.g.) tortoise git https://code.google.com/p/tortoisegit/ * Launch cmake GUI client . * Set 'Where is the source code:' text box and 'Where to build the binaries:' text box. * Click 'Configure' button. * Choose your Visual Studio version. * Click 'Generate' button. * Open the created msgpack.sln on Visual Studio. * Build all. Documentation You can get additional information including the tutorial on the wiki . CONTRIBUTING msgpack-c is developed on GitHub at msgpack/msgpack-c . To report an issue or send a pull request, use the issue tracker . Here's the list of great contributors . LICENSE msgpack-c is licensed under the Boost Software License, Version 1.0. See the LICENSE_1_0.txt file for details. MSGPACK/MSGPACK-OCAML HTTPS://GITHUB.COM/MSGPACK/MSGPACK-OCAML MSGPACK FOR OCAML BULID $ make $ sudo make install EXAMPLE Serialize/Deserialize for Msgpack Object (* serialize *) let bytes = Msgpack.Serialize.serialize_string (`FixArray ) (* deserialize *) let obj = Msgpack.Serialize.deserialize_string bytes Serialize/Deserialize for OCaml types (with meta_conv) open Msgpack_conv type t = { int : int; str : string; } with conv(msgpack) (* serialize *) let bytes = Msgpack.Serialize.serialize_string (msgpack_of_t { int = 42; str = "ans" }) (* deserialize *) let obj = t_of_msgpack (Msgpack.Serialize.deserialize_string bytes) See also, examlpe/ DEVELOPMENT Setup development enviroment with docker: $ docker-compose build $ docker-compose run app TEST $ ocaml setup.ml -configure --enable-tests $ make test PROOF If you want to use msgpack at OCaml, you need not do this section. This section for user intrested in formal verification. You need Coq 8.4 and omake. $ cd proof $ make $ cp *.ml* ../lib/core MSGPACK/MSGPACK-SMALLTALK HTTPS://GITHUB.COM/MSGPACK/MSGPACK-SMALLTALK MSGPACK-SMALLTALK MessagePack serialization library for various Smalltalk dialects. * Squeak * Pharo * VisualWorks * VA Smalltalk * Dolphin Smalltalk * GNU Smalltalk (Beta) * Cuis Sources are put as Cypress for the neutral accesses from various Smalltalk dialects. HOW TO USE Serialization MpMessagePack pack: or: messagePacked Deserialization MpMessagePack unpack: msgpackBytes or: Object fromMessagePack: msgBytes Samples map := Dictionary new. map at: 'someArray' asByteArray put: #(1 2.2 #). packed := map messagePacked. (Object fromMessagePack: packed) inspect. writeStream := WriteStream on: ByteArray new. encoder := MpEncoder on: writeStream. encoder nextPut: 1. encoder nextPut: #(2 3). dic := Dictionary new. dic at: 4 put: 5. encoder nextPut: dic. encoder nextPut: 'four' asByteArray. bytes := encoder contents. readStream := ReadStream on: bytes. decoder := MpDecoder on: readStream. whileFalse: How to install Please read HowToInstall.md . Loading the latest development version Squeak Installer squeaksource project: 'MessagePack'; install: 'ConfigurationOfMessagePack'. (Smalltalk at: #ConfigurationOfMessagePack) project development load Pharo Gofer it smalltalkhubUser: 'MasashiUmezawa' project: 'MessagePack'; configuration; load. (Smalltalk at: #ConfigurationOfMessagePack) project development load You might need MpTypeMapper initializeAll on new encoder/decoder-related updates. LOTEIXEIRA/AS3-MSGPACK HTTPS://GITHUB.COM/LOTEIXEIRA/AS3-MSGPACK MESSAGEPACK FOR ACTIONSCRIPT3 (FLASH, FLEX AND AIR). as3-msgpack was designed to work with the interfaces IDataInput and IDataOutput, thus the API might be easily connected with the native classes that handle binary data (such as ByteArray, Socket, FileStream and URLStream). Moreover, as3-msgpack is capable of decoding data from binary streams. Get started: http://loteixeira.github.io/lib/2013/08/19/as3-msgpack/ BASIC USAGE (ENCODING/DECODING): // create messagepack object var msgpack:MsgPack = new MsgPack(); // encode an array var bytes:ByteArray = msgpack.write(); // rewind the buffer bytes.position = 0; // print the decoded object trace(msgpack.read(bytes)); For downloads, source code and further information, check the project repository: https://github.com/loteixeira/as3-msgpack. MSGPACK/MSGPACK-PHP HTTPS://GITHUB.COM/MSGPACK/MSGPACK-PHP MSGPACK FOR PHP This extension provide API for communicating with MessagePack serialization. MessagePack is a binary-based efficient object serialization library. It enables to exchange structured objects between many languages like JSON. But unlike JSON, it is very fast and small. REQUIREMENT * PHP 5.0 + INSTALL Install from PECL Msgpack is an PECL extension, thus you can simply install it by: pecl install msgpack Compile Msgpack from source $/path/to/phpize $./configure $make && make install Example $data = array(0=>1,1=>2,2=>3); $msg = msgpack_pack($data); $data = msgpack_unpack($msg); ?> RESOURCES * msgpack VMIHAILENCO/MSGPACK HTTP://GODOC.ORG/GITHUB.COM/VMIHAILENCO/MSGPACK#PKG-EXAMPLES MESSAGEPACK ENCODING FOR GOLANG Supports: * Primitives, arrays, maps, structs, time.Time and interface{}. * Appengine *datastore.Key and datastore.Cursor. * CustomEncoder /CustomDecoder interfaces for custom encoding. * Extensions to encode type information. * Renaming fields via msgpack:"my_field_name". * Omitting individual empty fields via msgpack:",omitempty" tag or all empty fields in a struct . * Map keys sorting . * Encoding/decoding all structs as arrays or individual structs . * Encoder.UseJSONTag with Decoder.UseJSONTag can turn msgpack into drop-in replacement for JSON. * Simple but very fast and efficient queries . API docs: https://godoc.org/github.com/vmihailenco/msgpack. Examples: https://godoc.org/github.com/vmihailenco/msgpack#pkg-examples. INSTALLATION Install: go get -u github.com/vmihailenco/msgpack QUICKSTART func ExampleMarshal() { type Item struct { Foo string } b, err := msgpack.Marshal(&Item{Foo: "bar"}) if err != nil { panic(err) } var item Item err = msgpack.Unmarshal(b, &item) if err != nil { panic(err) } fmt.Println(item.Foo) // Output: bar } BENCHMARK BenchmarkStructVmihailencoMsgpack-4 200000 12814 ns/op 2128 B/op 26 allocs/op BenchmarkStructUgorjiGoMsgpack-4 100000 17678 ns/op 3616 B/op 70 allocs/op BenchmarkStructUgorjiGoCodec-4 100000 19053 ns/op 7346 B/op 23 allocs/op BenchmarkStructJSON-4 20000 69438 ns/op 7864 B/op 26 allocs/op BenchmarkStructGOB-4 10000 104331 ns/op 14664 B/op 278 allocs/op HOWTO Please go through examples to get an idea how to use this package. SEE ALSO * Golang PostgreSQL ORM * Golang message task queue MNEUMANN/RUST-MSGPACK HTTPS://GITHUB.COM/MNEUMANN/RUST-MSGPACK RUST-MSGPACK Msgpack implementation for Rust language. INSTALLATION Simply include the rust-msgpack in your Cargo dependencies. git = "git@github.com:mneumann/rust-msgpack.git" QUICKSTART extern crate msgpack; fn main() { let arr = vec!; let str = msgpack::Encoder::to_msgpack(&arr).ok().unwrap(); println!("Encoded: {}", str); let dec: Vec = msgpack::from_msgpack(str).ok().unwrap(); println!("Decoded: {}", dec); } To enable your own data structures to be automatically serialized from and to msgpack, derive from Encodable and Decodable as shown in the following example: extern crate serialize; # struct MyStruct { a: Vec, s: String } TESTING cargo test LICENSE This code licensed under the same terms as Rust itself: dual MIT/Apache2 license options. MURURU/MSGPACK-ELIXIR HTTPS://GITHUB.COM/MURURU/MSGPACK-ELIXIR MESSAGEPACK FOR ELIXIR INSTALLATION Add :message_pack as a dependency in your mix.exs file. defp deps do end USAGE # pack MessagePack.pack() #=> { :ok, <<147,1,2,3>> } MessagePack.pack!() #=> <<147,1,2,3>> # unpack MessagePack.unpack(<<147,1,2,3>>) #=> { :ok, } MessagePack.unpack!(<<147,1,2,3>>) #=> # unpack_once MessagePack.unpack_once(<<147,1,2,3,4>>) #=> {:ok, {, <<4>>}} MessagePack.unpack_once!(<<147,1,2,3,4>>) #=> {, <<4>>} OPTIONS * enable_string Support string type. This options is false by default. iex(1)> { :ok, bin } = MessagePack.pack(<<255>>) {:ok, <<161, 255>>} iex(3)> MessagePack.unpack(<<161, 255>>) {:ok, <<255>>} iex(4)> MessagePack.unpack(<<161, 255>>, enable_string: true) {:error, {:invalid_string, <<255>>}} * ext Support extention type. See test/message_pack_ext_test.exs. LICENSE MIT KTAKASHI/R6RS-MSGPACK HTTPS://GITHUB.COM/KTAKASHI/R6RS-MSGPACK MESSAGEPACK FOR R6RS SCHEME This is an implementation of MessagePack for R6RS Scheme. API REFERENCES _Function_ (pack! bv message) _Function_ (pack! bv message offset) Pack _message_ to message pack format bytevector and put it into the _bv_ destructively. Given _bv_ must have enough length to hold the message. Optional argument _offset_ indicates where to start with, default is 0. _Function_ (pack message) The same as PACK! but this one creates a new bytevector. _Function_ (pack-size message) Calculate the converted message size. _Function_ (unpack bv) _Function_ (unpack bv offset) Unpack the given message format bytevector to Scheme object. Optional argument _offset_ indicates where to start with, default is 0. _Function_ (get-unpack in) Unpack the given binary input port to Scheme object. CONVERSION RULES As you already know, Scheme doesn't have static types so the conversion of Scheme objects to message pack data might cause unexpected results. To avoid it, I will describe how conversion works. Scheme to message pack Integer conversion The library automatically decides proper size. More specifically, if it can fit to message pack's fixnum then library uses it, so are uint8-64. If the number is too big, then an error is raised. Users must know it tries to use uint as much as possible. If the given number was negative then sint will be used. Floating point conversion Unfortunately R6RS doesn't have difference between float and double. So when flonum is given then it always converts to double number. Collection conversion Message pack has collections which are map and array. And these are associated with alist (association list) and vector respectively. When you want to convert alist to message pack data, then you need to make sure the _cdr_ part will be the data and if you put _(("key" "value))__ then it will be converted to nested map. The collection size calculation is done automatically. It tries to use the smallest size. Message pack to Scheme The other way around is easy, it can simply restore the byte data to Scheme object. Following describes the conversion rules; Positive fixnum -> integer Negative fixnum -> integer uint8, uint16, uint32, uint64 -> integer sint8, sint16, sint32, sint64 -> integer Map -> alist Array -> vector fixstr, str8, str16, str32 -> string bit8, bit16, bit32 -> bytevector TESTED SCHEME IMPLEMENTATIONS Sagittarius Scheme Mosh Scheme Ypsilon Your contributions are always welcome. TODO * More testing * Extended type handling LICHTSO/NETLINK HTTPS://GITHUB.COM/LICHTSO/NETLINK MSGPACK V5 IMPLEMENTATION FOR C++ 11 FEATURES * std::streambuf serializer and deserializer * hierarchy or token stream * push and pull parser * byte wise data flow control SMALL EXAMPLE MsgPack::Serializer serializer(socket); std::vector> arrayWithoutElements, arrayWith3Elements; arrayWith3Elements.push_back(MsgPack::Factory(true)); arrayWith3Elements.push_back(MsgPack__Factory(Array(std::move(arrayWithoutElements)))); arrayWith3Elements.push_back(MsgPack::Factory("Hello World!")); serializer << MsgPack__Factory(Array(std::move(arrayWith3Elements))); MsgPack::Deserializer deserializer(socket); deserializer.deserialize((std::unique_ptr parsed) { std::cout << "Parsed: " << *parsed << "\n"; return false; }, true); READ MORE Tutorial UGORJI/GO HTTPS://GITHUB.COM/UGORJI/GO MESSAGEPACK AND BINC CODEC FOR GO LANGUAGE. _A High Performance, Feature-Rich, Idiomatic encode/decode and rpc library_. To install: go get github.com/ugorji/go/codec Source: Online documentation: Typical usage: // create and use decoder/encoder var ( v interface{} // value to decode/encode into r io.Reader w io.Writer b byte mh codec.MsgpackHandle ) dec = codec.NewDecoder(r, &mh) dec = codec.NewDecoderBytes(b, &mh) err = dec.Decode(&v) enc = codec.NewEncoder(w, &mh) enc = codec.NewEncoderBytes(&b, &mh) err = enc.Encode(v) //RPC Server go func() { for { conn, err := listener.Accept() rpcCodec := codec.GoRpc.ServerCodec(conn, h) //OR rpcCodec := codec.MsgpackSpecRpc.ServerCodec(conn, h) rpc.ServeCodec(rpcCodec) } }() //RPC Communication (client side) conn, err = net.Dial("tcp", "localhost:5555") rpcCodec := codec.GoRpc.ClientCodec(conn, h) //OR rpcCodec := codec.MsgpackSpecRpc.ClientCodec(conn, h) client := rpc.NewClientWithCodec(rpcCodec) SUZUKAZE/MRUBY-MSGPACK MESSAGEPACK FOR MRUBY MRUBY-MSGPACK WELCOME TO MESSAGEPACK FOR MRUBY MessagePack for mruby is MessagePack implimented in mruby language. msg = .to_msgpack #=> "\x93\x01\x02\x03" MessagePack.unpack(msg) #=> This is early vesion. Please check the methods that work in test folder. PLATFORM I test MessagePack for mruby in mac OSX 10.8.4. In the future it will work in Windows and Linux OS. GETTING STARTED * Download MessagePack for mruby at the command prompt: git clone https://github.com/suzukaze/mruby-msgpack.git * Add config.gem line to build_config.rb MRuby::Build.new do |conf| # ...(snip)... conf.gem :git => 'https://github.com/suzukaze/mruby-msgpack.git' end * Test at the command prompt: rake test * Build at the command prompt: rake MSGPACK-RUBY COMMIT mruby-msgpack is based on msgpack-ruby (a7c2dc34ef69c9132167e38009baa8420c460c9b) CONTRIBUTING I encourage you to contribute to MessagePack for mruby! LICENSE Author : Jun Hiroe Copyrigh : Copyright (c) 2013 Jun Hiroe License : MIT License LEXMAG/MSGPAX HTTPS://MSGPACK.ORG MSGPAX This library provides an API for serializing and de-serializing Elixir terms using the MessagePack format. Documentation is available online . FEATURES * Packing and unpacking Elixir terms via Msgpax.pack/1 and Msgpax.unpack/1 (and their ! bang variants). * Unpacking of partial slices of MessagePack-encoded terms via Msgpax.unpack_slice/1 . * Support for "Binary" and "Extension" MessagePack types via Msgpax.Bin and Msgpax.Ext , respectively. * Protocol-based packing through the Msgpax.Packer protocol, that can be derived for user-defined structs. * A Plug.Parsers plug (Msgpax.PlugParser ) to parse requests with MessagePack-encoded bodies. A detailed table that shows the relationship between Elixir types and MessagePack types can be found in the documentation for the Msgpax module . INSTALLATION Add :msgpax as a dependency in your mix.exs file: def deps do end Then, run mix deps.get in your shell to fetch the new dependency. LICENSE Msgpax is released under the ISC license . JINGWENO/MSGPACK_RAILS HTTPS://GITHUB.COM/JINGWENO/MSGPACK_RAILS MESSAGEPACK FOR RAILS The Rails way to serialize/deserialize objects with Message Pack . It implements the ActiveSupport encoder & decoder and the ActiveModel serializer for Message Pack. INSTALLATION Add this line to your application's Gemfile: gem 'msgpack_rails' And then execute: $ bundle Or install it yourself as: $ gem install msgpack_rails USAGE msgpack_rails converts data type using as_json before feeding it into msgpack . Here are a few examples: $ ActiveSupport::MessagePack.encode(:a => :b) => "\x81\xA1a\xA1b" $ ActiveSupport::MessagePack.encode(Time.now) => "\xB92013-09-11T10:40:39-07:00" $ Time.now.as_msgpack => "2013-09-11T10:48:13-07:00" $ Time.now.to_msgpack => "\xB92013-09-11T10:40:39-07:00" $ ActiveSupport::MessagePack.decode Time.now.to_msgpack => "2013-09-11T11:23:07-07:00" # After setting ActiveSupport.parse_msgpack_times to true $ ActiveSupport::MessagePack.decode Time.now.to_msgpack => Wed, 11 Sep 2013 11:25:18 -0700 You can also use it as part of ActiveModel, similar to to_json: class Contact include ActiveModel::Serializers::MessagePack ... end @contact = Contact.new @contact.name = 'Owen Ou' @contact.age = 28 @contact.created_at = Time.utc(2006, 8, 1) @contact.awesome = true @contact.preferences = { 'shows' => 'anime' } @contact.to_msgpack # => msgpack output @contact.to_msgpack(:root => true) # => include root in msgpack output CONTRIBUTING * Fork it * Create your feature branch (git checkout -b my-new-feature) * Commit your changes (git commit -am 'Add some feature') * Push to the branch (git push origin my-new-feature) * Create new Pull Request VSERGEEV/U-MSGPACK-PYTHON HTTPS://GITHUB.COM/VSERGEEV/U-MSGPACK-PYTHON U-MSGPACK-PYTHON u-msgpack-python is a lightweight MessagePack serializer and deserializer module written in pure Python, compatible with both Python 2 and 3, as well CPython and PyPy implementations of Python. u-msgpack-python is fully compliant with the latest MessagePack specification . u-msgpack-python is currently distributed on PyPI: https://pypi.python.org/pypi/u-msgpack-python and as a single file: umsgpack.py INSTALLATION With pip: $ pip install u-msgpack-python With easy_install: $ easy_install u-msgpack-python or simply drop umsgpack.py into your project! $ wget https://raw.github.com/vsergeev/u-msgpack-python/master/umsgpack.py EXAMPLES Basic Example: >>> import umsgpack >>> umsgpack.packb({u"compact": True, u"schema": 0}) b'\x82\xa7compact\xc3\xa6schema\x00' >>> umsgpack.unpackb(_) {u'compact': True, u'schema': 0} >>> A more complicated example: >>> umsgpack.packb( ... ) b'\x97\x01\xc3\xc2\xce\xff\xff\xff\xff\x82\xa3foo\xc4\x03\x80\x01\ \x02\xa3bar\x94\x01\x02\x03\x81\xa1a\x94\x01\x02\x03\x80\xff\xcb\ @\x00\xfc\xd3Z\x85\x87\x94' >>> umsgpack.unpackb(_) >>> Streaming serialization with file-like objects: >>> f = open('test.bin', 'wb') >>> umsgpack.pack({u"compact": True, u"schema": 0}, f) >>> umsgpack.pack(, f) >>> f.close() >>> >>> f = open('test.bin', 'rb') >>> umsgpack.unpack(f) {u'compact': True, u'schema': 0} >>> umsgpack.unpack(f) >>> f.close() >>> Serializing and deserializing a raw Ext type: >>> # Create an Ext object with type 0x05 and data b"\x01\x02\x03" ... foo = umsgpack.Ext(0x05, b"\x01\x02\x03") >>> umsgpack.packb({u"stuff": foo, u"awesome": True}) b'\x82\xa5stuff\xc7\x03\x05\x01\x02\x03\xa7awesome\xc3' >>> >>> bar = umsgpack.unpackb(_) >>> print(bar) Ext Object (Type: 0x05, Data: 0x01 0x02 0x03) >>> bar.type 5 >>> bar.data b'\x01\x02\x03' >>> Serializing and deserializing application-defined types with Ext handlers: >>> umsgpack.packb(, ... ext_handlers = { ... complex: lambda obj: ... umsgpack.Ext(0x30, struct.pack("ff", obj.real, obj.imag)), ... decimal.Decimal: lambda obj: ... umsgpack.Ext(0x40, str(obj).encode()), ... }) b'\x92\xd70\x00\x00\x80?\x00\x00\x00@\xd6@0.31' >>> umsgpack.unpackb(_, ... ext_handlers = { ... 0x30: lambda ext: ... complex(*struct.unpack("ff", ext.data)), ... 0x40: lambda ext: ... decimal.Decimal(ext.data.decode()), ... }) >>> Python standard library style names dump, dumps, load, loads are also available: >>> umsgpack.dumps({u"compact": True, u"schema": 0}) b'\x82\xa7compact\xc3\xa6schema\x00' >>> umsgpack.loads(_) {u'compact': True, u'schema': 0} >>> >>> f = open('test.bin', 'wb') >>> umsgpack.dump({u"compact": True, u"schema": 0}, f) >>> f.close() >>> >>> f = open('test.bin', 'rb') >>> umsgpack.load(f) {u'compact': True, u'schema': 0} >>> MORE INFORMATION See the project page for more information on options, exceptions, behavior, and testing. LICENSE u-msgpack-python is MIT licensed. See the included LICENSE file for more details. MARKSTINSON/LUA-MESSAGEPACK HTTP://FPERRAD.GITHUB.IO/LUA-MESSAGEPACK/ LUA-MESSAGEPACK : A PURE LUA IMPLEMENTATION (SPEC V5) INTRODUCTION MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON but it's faster and smaller. It's a pure Lua implementation, without dependency. And it's really fast with LuaJIT . LINKS The homepage is at http://fperrad.github.io/lua-MessagePack/ , and the sources are hosted at http://github.com/fperrad/lua-MessagePack/ . COPYRIGHT AND LICENSE Copyright (c) 2012-2013 Francois Perrad This library is licensed under the terms of the MIT/X11 license, like Lua itself. JULIAIO/MSGPACK.JL HTTPS://GITHUB.COM/JULIAIO/MSGPACK.JL MSGPACK Provides basic support for the msgpack format. julia> import MsgPack julia> MsgPack.pack("hi") 3-element Array{Uint8,1}: 0xa2 0x68 0x69 julia> a = MsgPack.pack() 6-element Array{Uint8,1}: 0x93 0x01 0x02 0xa2 0x68 0x69 julia> MsgPack.unpack(MsgPack.pack(4.5)) 4.5 julia> f = open("in.mp") julia> MsgPack.unpack(f) "hello" julia> f2 = open("out.mp", "w") julia> MsgPack.pack(f2, ) NOTE: The standard method for encoding integers in msgpack is to use the most compact representation possible, and to encode negative integers as signed ints and non-negative numbers as unsigned ints. For compatibility with other implementations, I'm following this convention. On the unpacking side, every integer type becomes an Int64 in Julia, unless it doesn't fit (ie. values greater than 2^63 are unpacked as Uint64). I might change this at some point, and/or provide a way to control the unpacked types. The Extension Type The MsgPack spec defines the extension type to be a tuple of (typecode, bytearray) where typecode is an application-specific identifier for the data in bytearray. MsgPack.jl provides support for the extension type through the Ext immutable. It is defined like so struct Ext typecode::Int8 data::Vector{Uint8} end and used like this julia> a = 6-element Array{UInt8,1}: 0x34 0xff 0x76 0x22 0xd3 0xab julia> b = Ext(22, a) MsgPack.Ext(22,UInt8) julia> p = pack(b) 9-element Array{UInt8,1}: 0xc7 0x06 0x16 0x34 0xff 0x76 0x22 0xd3 0xab julia> c = unpack(p) MsgPack.Ext(22,UInt8) julia> c == b true MsgPack reserves typecodes in the range for future types specified by the MsgPack spec. MsgPack.jl enforces this when creating an Ext but if you are packing an implementation defined extension type (currently there are none) you can pass impltype=true. julia> Ext(-43, Uint8) ERROR: MsgPack Ext typecode -128 through -1 reserved by implementation in call at /Users/sean/.julia/v0.4/MsgPack/src/MsgPack.jl:48 julia> Ext(-43, Uint8, impltype=true) MsgPack.Ext(-43,UInt8) Serialization MsgPack.jl also defines the extserialize and extdeserialize convenience functions. These functions can turn an arbitrary object into an Ext and vice-versa. julia> mutable struct Point{T} x::T y::T end julia> r = Point(2.5, 7.8) Point{Float64}(2.5,7.8) julia> e = MsgPack.extserialize(123, r) MsgPack.Ext(123,UInt8) julia> s = MsgPack.extdeserialize(e) (123,Point{Float64}(2.5,7.8)) julia> s Point{Float64}(2.5,7.8) julia> r Point{Float64}(2.5,7.8) Since these functions use serialize under the hood they are subject to the following caveat. > In general, this process will not work if the reading and writing > are done by different versions of Julia, or an instance of Julia > with a different system image. EDMA2/CLOJURE-MSGPACK HTTPS://GITHUB.COM/EDMA2/CLOJURE-MSGPACK CLOJURE-MSGPACK clojure-msgpack is a lightweight and simple library for converting between native Clojure data structures and MessagePack byte formats. clojure-msgpack only depends on Clojure itself; it has no third-party dependencies. INSTALLATION USAGE Basic * pack: Serialize object as a sequence of java.lang.Bytes. * unpack Deserialize bytes as a Clojure object. (require ') (require 'msgpack.clojure-extensions) (msg/pack {:compact true :schema 0}) ; => # (msg/unpack (msg/pack {:compact true :schema 0})) ; => {:schema 0, :compact true} Streaming clojure-msgpack provides a streaming API for situations where it is more convenient or efficient to work with byte streams instead of fixed byte arrays (e.g. size of object is not known ahead of time). The streaming counterpart to msgpack.core/pack is msgpack.core/pack-stream which returns nil and accepts either java.io.OutputStream or java.io.DataOutput as an additional argument. msgpack.core/unpack is in "streaming mode" when the argument is of type java.io.DataInput or java.io.InputStream . (use 'clojure.java.io) (with-open (msg/pack-stream {:compact true :schema 0} s)) (with-open (msg/unpack s)) ; => {:schema 0, :compact true} Core types CLOJURE MESSAGEPACK nil Nil java.lang.Boolean Boolean java.lang.Byte Integer java.lang.Short Integer java.lang.Integer Integer java.lang.Long Integer java.lang.BigInteger Integer clojure.lang.BigInt Integer java.lang.Float Float java.lang.Double Float java.math.BigDecimal Float java.lang.String String clojure.lang.Sequential Array clojure.lang.IPersistentMap Map msgpack.core.Ext Extended Serializing a value of unrecognized type will fail with IllegalArgumentException. See Application types if you want to register your own types. Clojure types Some native Clojure types don't have an obvious MessagePack counterpart. We can serialize them as Extended types. To enable automatic conversion of these types, load the clojure-extensions library. CLOJURE MESSAGEPACK clojure.lang.Keyword Extended (type = 3) clojure.lang.Symbol Extended (type = 4) java.lang.Character Extended (type = 5) clojure.lang.Ratio Extended (type = 6) clojure.lang.IPersistentSet Extended (type = 7) With msgpack.clojure-extensions: (require 'msgpack.clojure-extensions) (msg/pack :hello) ; => # Without msgpack.clojure-extensions: (msg/pack :hello) ; => IllegalArgumentException No implementation of method: :pack-stream of ; protocol: #'msgpack.core/Packable found for class: clojure.lang.Keyword ; clojure.core/-cache-protocol-fn (core _deftype.clj:544) Application types You can also define your own Extended types with extend-msgpack. (require ') (defrecord Person ) (extend-msgpack Person 100 (.getBytes (:name p)) (->Person (String. bytes))) (msg/unpack (msg/pack )) ; => (#user.Person{:name "bob"} 5 "test") Options All pack and unpack functions take an optional map of options: * :compatibility-mode Serialize/deserialize strings and bytes using the raw-type defined here: https://github.com/msgpack/msgpack/blob/master/spec-old.md Note: No error is thrown if an unpacked value is reserved under the old spec but defined under the new spec. We always deserialize something if we can regardless of compatibility-mode. (msg/pack (byte-array (byte 9)) {:compatibility-mode true}) LICENSE clojure-msgpack is MIT licensed. See the included LICENSE file for more details. TKOB/MLMSGPACK HTTPS://GITHUB.COM/TKOB/MLMSGPACK ML-MESSAGEPACK MessagePack implementation for Standard ML (SML) FEATURES * Portable: Depends only on the required components of the SML Basis Library specification. * Composable: Composable combinators for encoding and decoding. USAGE MLton and MLKit Include mlmsgpack.mlb in your MLB file. Poly/ML From the interactive shell, use .sml files in the following order. * mlmsgpack-aux.sml * realprinter-default.sml * mlmsgpack.sml SML/NJ Use mlmsgpack.cm. Moscow ML From the interactive shell, use .sml files in the following order. * large.sml * mlmsgpack-aux.sml * realprinter-fail.sml * mlmsgpack.sml Makefile.mosml is also provided. HaMLet From the interactive shell, use .sml files in the following order. * mlmsgpack-aux.sml * realprinter-fail.sml * mlmsgpack.sml Alice ML Makefile.alice is provided. make -f Makefile.alice alicerun mlmsgpack-test SML# For separate compilation, .smi files are provided. Require mlmsgpack.smi from your .smi file. From the interactive shell, use .sml files in the following order. * mlmsgpack-aux.sml * realprinter-default.sml * mlmsgpack.sml TUTORIAL See TUTORIAL.md. KNOWN PROBLEMS Our recommendation is MLton, MLKit, Poly/ML and SML#(>=2.0.0) as all tests passed on them. SML/NJ and Moscow ML are fine if you don't use real values. SML/NJ Packing real values fail or produces imprecise results in some cases. Moscow ML Packing real values is not supported, since some components of the SML Basis Library are not provided. HaMLet Packing real values is not supported, since some components of the SML Basis Library are not provided. Some functions are very slow, although they work properly. (We tested HaMLet compiled with MLton.) Alice ML Packing real values is not supported, since some components of the SML Basis Library are not provided. Also, some unit tests fail. SML# Most functions do not work properly because of bugs of SML# prior to version 2.0.0. SEE ALSO There already exists another MessagePack implemenatation for SML, called MsgPack-SML, which is targeted for MLton. https://msgpacksml.codeplex.com/ ML-MessagePack is written from scratch and not a fork of MsgPack-SML. For information on MessagePack, see: http://msgpack.org/ CAMGUNZ/CMP HTTPS://GITHUB.COM/CAMGUNZ/CMP CMP CMP is a C implementation of the MessagePack serialization format. It currently implements version 5 of the MessagePack Spec . CMP's goal is to be lightweight and straightforward, forcing nothing on the programmer. LICENSE While I'm a big believer in the GPL, I license CMP under the MIT license. EXAMPLE USAGE The following examples use a file as the backend, and are modeled after the examples included with the msgpack-c project. #include #include #include #include #include "cmp.h" static bool read_bytes(void *data, size_t sz, FILE *fh) { return fread(data, sizeof(uint8_t), sz, fh) == (sz * sizeof(uint8_t)); } static bool file_reader(cmp_ctx_t *ctx, void *data, size_t limit) { return read_bytes(data, limit, (FILE *)ctx->buf); } static bool file_skipper(cmp_ctx_t *ctx, size_t count) { return fseek((FILE *)ctx->buf, count, SEEK_CUR); } static size_t file_writer(cmp_ctx_t *ctx, const void *data, size_t count) { return fwrite(data, sizeof(uint8_t), count, (FILE *)ctx->buf); } void error_and_exit(const char *msg) { fprintf(stderr, "%s\n\n", msg); exit(EXIT_FAILURE); } int main(void) { FILE *fh = NULL; cmp_ctx_t cmp; uint32_t array_size = 0; uint32_t str_size = 0; char hello = {0, 0, 0, 0, 0, 0}; char message_pack = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}; fh = fopen("cmp_data.dat", "w+b"); if (fh == NULL) error_and_exit("Error opening data.dat"); cmp_init(&cmp, fh, file_reader, file_skipper, file_writer); if (!cmp_write_array(&cmp, 2)) error_and_exit(cmp_strerror(&cmp)); if (!cmp_write_str(&cmp, "Hello", 5)) error_and_exit(cmp_strerror(&cmp)); if (!cmp_write_str(&cmp, "MessagePack", 11)) error_and_exit(cmp_strerror(&cmp)); rewind(fh); if (!cmp_read_array(&cmp, &array_size)) error_and_exit(cmp_strerror(&cmp)); /* You can read the str byte size and then read str bytes... */ if (!cmp_read_str_size(&cmp, &str_size)) error_and_exit(cmp_strerror(&cmp)); if (str_size > (sizeof(hello) - 1)) error_and_exit("Packed 'hello' length too long\n"); if (!read_bytes(hello, str_size, fh)) error_and_exit(cmp_strerror(&cmp)); /* * ...or you can set the maximum number of bytes to read and do it all in * one call */ str_size = sizeof(message_pack); if (!cmp_read_str(&cmp, message_pack, &str_size)) error_and_exit(cmp_strerror(&cmp)); printf("Array Length: %u.\n", array_size); printf("\n", hello, message_pack); fclose(fh); return EXIT_SUCCESS; } ADVANCED USAGE See the examples folder. FAST, LIGHTWEIGHT, FLEXIBLE, AND ROBUST CMP uses no internal buffers; conversions, encoding and decoding are done on the fly. CMP's source and header file together are ~4k LOC. CMP makes no heap allocations. CMP uses standardized types rather than declaring its own, and it depends only on stdbool.h, stdint.h and string.h. CMP is written using C89 (ANSI C), aside, of course, from its use of fixed-width integer types and bool. On the other hand, CMP's test suite requires C99. CMP only requires the programmer supply a read function, a write function, and an optional skip function. In this way, the programmer can use CMP on memory, files, sockets, etc. CMP is portable. It uses fixed-width integer types, and checks the endianness of the machine at runtime before swapping bytes (MessagePack is big-endian). CMP provides a fairly comprehensive error reporting mechanism modeled after errno and strerror. CMP is thread aware; while contexts cannot be shared between threads, each thread may use its own context freely. CMP is tested using the MessagePack test suite as well as a large set of custom test cases. Its small test program is compiled with clang using -Wall -Werror -Wextra ... along with several other flags, and generates no compilation errors in either clang or GCC. CMP's source is written as readably as possible, using explicit, descriptive variable names and a consistent, clear style. CMP's source is written to be as secure as possible. Its testing suite checks for invalid values, and data is always treated as suspect before it passes validation. CMP's API is designed to be clear, convenient and unsurprising. Strings are null-terminated, binary data is not, error codes are clear, and so on. CMP provides optional backwards compatibility for use with other MessagePack implementations that only implement version 4 of the spec. BUILDING There is no build system for CMP. The programmer can drop cmp.c and cmp.h in their source tree and modify as necessary. No special compiler settings are required to build it, and it generates no compilation errors in either clang or gcc. VERSIONING CMP's versions are single integers. I don't use semantic versioning because I don't guarantee that any version is completely compatible with any other. In general, semantic versioning provides a false sense of security. You should be evaluating compatibility yourself, not relying on some stranger's versioning convention. STABILITY I only guarantee stability for versions released on the releases page . While rare, both master and develop branches may have errors or mismatched versions. BACKWARDS COMPATIBILITY Version 4 of the MessagePack spec has no BIN type, and provides no STR8 marker. In order to remain backwards compatible with version 4 of MessagePack, do the following: Avoid these functions: * cmp_write_bin * cmp_write_bin_marker * cmp_write_str8_marker * cmp_write_str8 * cmp_write_bin8_marker * cmp_write_bin8 * cmp_write_bin16_marker * cmp_write_bin16 * cmp_write_bin32_marker * cmp_write_bin32 Use these functions in lieu of their v5 counterparts: * cmp_write_str_marker_v4 instead of cmp_write_str_marker * cmp_write_str_v4 instead of cmp_write_str * cmp_write_object_v4 instead of cmp_write_object DANELLIS/DART-MSGPACK HTTPS://GITHUB.COM/DANELLIS/DART-MSGPACK DART-MSGPACK This is a very early release of my MessagePack library for Dart. Currently, message classes must be written by hand. For example: class NotificationFrame extends Message { String kind; Map data; NotificationFrame(this.kind, this.data); static NotificationFrame fromList(List f) => new NotificationFrame(f, f); List toList() => ; } For each class you need to define the fromList and toList methods, which convert from and to a list of fields respectively. For example usage, see the unit tests. REEZE/MSGPACK-HHVM HTTPS://GITHUB.COM/REEZE/MSGPACK-HHVM MSGPACK-HHVM Build Status: Msgpack for HHVM, It is a msgpack binding for HHVM API * msgpack_pack(mixed $input) : string; pack a input to msgpack, object and resource are not supported, array and other types supported, false on failure. * msgpack_unpack(string $pac) : mixed; unpack a msgpack. INSTALLATION $ git clone https://github.com/reeze/msgpack-hhvm --depth=1 $ cd msgpack-hhvm $ hphpize && cmake . && make $ cp msgpack.so /path/to/your/hhvm/ext/dir If you don't have hphpize program, please intall package hhvm-dev $ sudo apt-get install hhvm-dev CONTRIBUTION AND ISSUES Feel free to send Pull Requests for bug report at: http://github.com/reeze/msgpack-hhvm/issues AUTHORS * Reeze Xia reeze@php.net GAB-KM/MSGPACK-FSHARP HTTPS://GITHUB.COM/GAB-KM/MSGPACK-FSHARP MESSAGEPACK FOR F# WHAT IS THIS? MessagePack is a fast and compact binary serialization library. MessagePack for F# is a MessagePack implementation of F#, by F#, for F#. USAGE open MsgPack |> Array.map (Value.UInt8) |> Value.Array |> Packer.packOne //=> val it : byte = Unpacker.unpack //=> COPYRIGHT Copyright (c) 2014- Kazuhiro Matsushima LICENSE Distributed under the Apache License, Version 2.0 . KOMAMITSU/JACKSON-DATAFORMAT-MSGPACK HTTPS://GITHUB.COM/KOMAMITSU/JACKSON-DATAFORMAT-MSGPACK JACKSON-DATAFORMAT-MSGPACK THIS PROJECT IS MERGED TO MSGPACK-JAVA!! Yay! SEE MSGPACK-JAVA/MSGPACK-JACKSON FOR THE UPDATED DOCUMENTS OVERVIEW This Jackson extension library handles reading and writing of data encoded in MessagePack data format. It extends standard Jackson streaming API (JsonFactory, JsonParser, JsonGenerator), and as such works seamlessly with all the higher level data abstractions (data binding, tree model, and pluggable extensions). MAVEN DEPENDENCY To use this module on Maven-based projects, use following dependency: org.komamitsu jackson-dataformat-msgpack 0.0.3 USAGE Only thing you need to do is to instantiate MessagePackFactory and pass it to the constructor of ObjectMapper. ObjectMapper objectMapper = new ObjectMapper(new MessagePackFactory()); ExamplePojo orig = new ExamplePojo("komamitsu"); byte bytes = objectMapper.writeValueAsBytes(orig); ExamplePojo value = objectMapper.readValue(bytes, ExamplePojo.class); System.out.println(value.getName()); // => komamitsu Also, you can exchange data among multiple languages. Java // Serialize Map obj = new HashMap(); obj.put("foo", "hello"); obj.put("bar", "world"); byte bs = objectMapper.writeValueAsBytes(obj); // bs => Ruby require 'msgpack' # Deserialize xs = MessagePack.unpack(xs.pack("C*")) # => {"foo"=>"hello", "bar"=>"world"} # Serialize .to_msgpack.unpack('C*') # => Java // Deserialize bs = new byte {(byte) 148, (byte) 164, 122, 101, 114, 111, 1, (byte) 203, 64, 0, 0, 0, 0, 0, 0, 0, (byte) 192}; TypeReference> typeReference = new TypeReference>(){}; List xs = objectMapper.readValue(bs, typeReference); // xs => BRIANDW/SWIFTPACK HTTPS://GITHUB.COM/BRIANDW/SWIFTPACK SWIFTPACK SwiftPack is MessagePack packer and unpacker written almost entirely in Swift. ISSUES * There are few Objective-C dependencies, mostly NSData. I hope to remove these dependencies as Swift matures. * Maps currently only support String types for keys. COPYRIGHT AND LICENSE Copyright (c) 2014 Brian Williams This software is licensed under the terms of the MIT license. GABRIEL/MPMESSAGEPACK HTTPS://GITHUB.COM/GABRIEL/MPMESSAGEPACK INSTALL pod "MPMessagePack" WRITING #import NSDictionary *dict = @{ @"n": @(32134123), @"bool": @(YES), @"array": @, @"body": , }; NSData *data = ; Or via MPMessagePackWriter. NSError *error = nil; NSData *data = ; If you need to use an ordered dictionary. MPOrderedDictionary *dict = ; ; ; ; READING id obj = ; MPMessagePackReader *reader = ; id obj1 = ; // Read an object id obj2 = ; // Read another object MCOLLINA/MSGPACK5 HTTPS://GITHUB.COM/MCOLLINA/MSGPACK5 MSGPACK5 A msgpack v5 implementation for node.js and the browser, with extension point support. INSTALL npm install msgpack5 --save USAGE var msgpack = require('msgpack5')() // namespace our extensions , a = new MyType(2, 'a') , encode = msgpack.encode , decode = msgpack.decode msgpack.register(0x42, MyType, mytipeEncode, mytipeDecode) console.log(encode({ 'hello': 'world' }).toString('hex')) // 81a568656c6c6fa5776f726c64 console.log(decode(encode({ 'hello': 'world' }))) // { hello: 'world' } console.log(encode(a).toString('hex')) // d5426161 console.log(decode(encode(a)) instanceof MyType) // true console.log(decode(encode(a))) // { value: 'a', size: 2 } function MyType(size, value) { this.value = value this.size = size } function mytipeEncode(obj) { var buf = new Buffer(obj.size) buf.fill(obj.value) return buf } function mytipeDecode(data) { var result = new MyType(data.length, data.toString('utf8', 0, 1)) , i for (i = 0; i < data.length; i++) { if (data.readUInt8(0) != data.readUInt8(i)) { throw new Error('should all be the same') } } return result } IN THE BROWSER This library is compatible with Browserify . If you want to use standalone, grab the file in the dist folder of this repo, and use in your own HTML page, the module will expose a msgpack5 global. To build npm run build API API * MSGPACK() * msgpack().ENCODE() * msgpack().DECODE() * msgpack().REGISTERENCODER() * msgpack().REGISTERDECODER() * msgpack().REGISTER() * msgpack().ENCODER() * msgpack().DECODER() ------------------------- msgpack(options(obj)) Creates a new instance on which you can register new types for being encoded. options: * forceFloat64, a boolean to that forces all floats to be encoded as 64-bits floats. Defaults to false. * compatibilityMode, a boolean that enables "compatibility mode" which doesn't use str 8 format. Defaults to false. * disableTimestampEncoding, a boolean that when set disables the encoding of Dates into the timestamp extension type . Defaults to false. ------------------------- encode(object) Encodes object in msgpack, returns a bl . ------------------------- decode(buf) Decodes buf from in msgpack. buf can be a Buffer or a bl instance. In order to support a stream interface, a user must pass in a bl instance. ------------------------- registerEncoder(check(obj), encode(obj)) Register a new custom object type for being automatically encoded. The arguments are: * check, a function that will be called to check if the passed object should be encoded with the encode function * encode, a function that will be called to encode an object in binary form; this function MUST return a Buffer which include the same type for registerDecoder. ------------------------- registerDecoder(type, decode(buf)) Register a new custom object type for being automatically decoded. The arguments are: * type, is a greater than zero integer identificating the type once serialized * decode, a function that will be called to decode the object from the passed Buffer ------------------------- register(type, constructor, encode(obj), decode(buf)) Register a new custom object type for being automatically encoded and decoded. The arguments are: * type, is a greater than zero integer identificating the type once serialized * constructor, the function that will be used to match the objects with instanceof * encode, a function that will be called to encode an object in binary form; this function MUST return a Buffer that can be deserialized by the decode function * decode, a function that will be called to decode the object from the passed Buffer This is just a commodity that calls registerEncoder and registerDecoder internally. ------------------------- encoder(options) Builds a stream in object mode that encodes msgpack. Supported options: * wrap, objects should be passed to encoder in wrapped object {value: data}. Wrap option should be used if you need to pass null to encoder. ------------------------- decoder(options) Builds a stream in object mode that decodes msgpack. Supported options: * wrap, decoded objects returned in wrapped object {value: data}. Wrap option should be used if stream contains msgpack nil. LEVELUP SUPPORT MSGPACK5 can be used as a LevelUp valueEncoding straight away: var level = require('level') , pack = msgpack() , db = level('foo', { valueEncoding: pack }) , obj = { my: 'obj' } db.put('hello', obj, function(err) { db.get('hello', function(err, result) { console.log(result) db.close() }) }) RELATED PROJECTS * msgpack5rpc : An implementation of the msgpack-rpc spec on top of this library. DISCLAIMER This library is built fully on JS and on bl to simplify the code. Every improvement that keeps the same API is welcome. ACKNOWLEDGEMENTS This project was kindly sponsored by nearForm . This library was originally built as the data format for JSChan . LICENSE MIT RODRIGOSETTI/MESSAGEPACK HTTP://HACKAGE.HASKELL.ORG/PACKAGE/MESSAGEPACK MESSAGEPACK FOR HASKELL This implementation defines an messagepack Object type, which is an instance of Serialize (from cereal ): data Object = ObjectNil | ObjectUInt Word64 | ObjectInt Int64 | ObjectBool Bool | ObjectFloat Float | ObjectDouble Double | ObjectString ByteString | ObjectBinary ByteString | ObjectArray | ObjectMap (M.Map Object Object ) | ObjectExt !Int8 BS.ByteString deriving (Eq, Ord, Show) instance Serialize Object where -- ... Thus, you can use cereal's encode and decode to pack and unpack objects. YMOFEN/MSGPACK-DELPHI HTTPS://GITHUB.COM/YMOFEN/MSGPACK-DELPHI MSGPACK FOR DELPHI It's like JSON but small and fast. unit Owner: D10.Mofen, qdac.swish contact: qq:185511468, email:185511468@qq.com welcome to report bug WORKS WITH * Delphi 7 (tested) * Delphi 2007 (tested) * XE5, XE6, XE7, FMX (tested) CHANGES: * first release 2014-08-15 13:05:13 * add array support 2014-08-19 12:18:47 * add andriod support 2014-09-08 00:45:27 * fixed int32, int64 parse bug< integer, int64 parse zero> 2014-11-09 22:35:27 * add EncodeToFile/DecodeFromFile 2014-11-13 12:30:58 * fix asVariant = null (thanks for cyw(26890954)) 2014-11-14 09:05:52 * fix AsInteger = -1 bug (thanks for cyw(26890954)) 2014-11-14 12:15:52 * fix AsInteger = -127 bug check int64/integer/cardinal/word/shortint/smallint/byte assign, encode,decode, read 2014-11-14 12:30:38 * fix AsFloat = 2.507182 bug thanks fo -芒果 1939331207 2014-11-21 12:37:04 * add AddArrayChild func 2015-03-25 17:47:28 * add remove/removeFromParent/Delete function 2015-08-29 22:37:48 Code Example var lvMsg, lvMsg2:TSimpleMsgPack; lvBytes:TBytes; s:string; begin lvMsg := TSimpleMsgPack.Create; lvMsg.S := '汉字,ascii'; if dlgOpen.Execute then begin lvMsg.S := ExtractFileName(dlgOpen.FileName); // file binary data lvMsg.ForcePathObject('key.image.data').LoadBinaryFromFile(dlgOpen.FileName); end; // lvBytes := lvMsg.EncodeToBytes; lvMsg2 := TSimpleMsgPack.Create; lvMsg2.DecodeFromBytes(lvBytes); // Memo1.Lines.Add(lvMsg2.S); if lvMsg2.S <> '' then begin s := ExtractFilePath(ParamStr(0)) + lvMsg2.S; Memo1.Lines.Add('file saved'); Memo1.Lines.Add(s); lvMsg2.ForcePathObject('key.image.data').SaveBinaryToFile(s); end; CHINAWSB/QMSGPACK-DELPHI HTTP://WWW.QDAC.CC QMsgPack-Messagepack for Delphi/C++ Builder QMsgPack is a simple and powerful Delphi & C++ Builder implementation for messagepack protocol. QMsgPack is a part of QDAC 3.0,Source code hosted in Sourceforge(http://sourceforge.net/p/qdac3 ). Feathers Full types support,include messagepack extension type Full open source,free for used in ANY PURPOSE Quick and simple interface RTTI support include Install QMsgPack is not a desgin time package.So just place QMsgPack files into search path and add to your project. Support Topic in Website (http://www.qdac.cc/?cat=44) ,CHINESE only Mail to author (chinawsb@sina.com) Post in forum (http://tieba.baidu.com/f?kw=qdac) QQ Group No：250530692 (http://jq.qq.com/?_wv=1027&k=ZH6mZR ) Source check out HTTP (http://svn.code.sf.net/p/qdac3/code/) SVN (svn://svn.code.sf.net/p/qdac3/code/) Example var lvMsg, lvMsg2:TQMsgPack; lvBytes:TBytes; s:string; begin lvMsg := TQMsgPack.Create; lvMsg.ForcePath('key.obj').AsString := '汉字,ascii'; // lvBytes := lvMsg.Encode; lvMsg2 := TQMsgPack.Create; lvMsg2.Parse(lvBytes); // showMessage(lvMsg.ForcePath('key.obj').AsString); .... ROMIXLAB/QMSGPACK HTTPS://GITHUB.COM/ROMIXLAB/QMSGPACK MESSAGEPACK FOR QT Full documentation is here mp.marsw.ru INSTALLATION Clone repository: git clone https://github.com/romixlab/qmsgpack.git cd qmsgpack mkdir build cd build cmake .. make install SAMPLE USAGE Packing QVariantList list; list << 1 << 2 << 3; QByteArray array = MsgPack::pack(list); Unpacking: QVariantList unpacked = MsgPack::unpack(array).toList(); Streaming API: // packing MsgPackStream stream(&ba, QIODevice::WriteOnly); stream << 1 << 2.3 << "some string"; // unpacking MsgPackStream stream(ba); int a; double b; QSting s; stream >> a >> b >> s; QT TYPES AND USER TYPES There is packers and unpackers for QColor, QTime, QDate, QDateTime, QPoint, QSize, QRect. Also you can create your own packer/unpacker methods for Qt or your own types. See docs for details. TINYLIB/MSGP HTTPS://GITHUB.COM/TINYLIB/MSGP MESSAGEPACK CODE GENERATOR This is a code generation tool and serialization library for MessagePack . You can read more about MessagePack in the wiki , or at msgpack.org . Why? * Use Go as your schema language * Performance * JSON interop * User-defined extensions * Type safety * Encoding flexibility Quickstart In a source file, include the following directive: //go:generate msgp The msgp command will generate serialization methods for all exported type declarations in the file. You can read more about the code generation options here . Use Field names can be set in much the same way as the encoding/json package. For example: type Person struct { Name string `msg:"name"` Address string `msg:"address"` Age int `msg:"age"` Hidden string `msg:"-"` // this field is ignored unexported bool // this field is also ignored } By default, the code generator will satisfy msgp.Sizer, msgp.Encodable, msgp.Decodable, msgp.Marshaler, and msgp.Unmarshaler. Carefully-designed applications can use these methods to do marshalling/unmarshalling with zero heap allocations. While msgp.Marshaler and msgp.Unmarshaler are quite similar to the standard library's json.Marshaler and json.Unmarshaler, msgp.Encodable and msgp.Decodable are useful for stream serialization. (*msgp.Writer and *msgp.Reader are essentially protocol-aware versions of *bufio.Writer and *bufio.Reader, respectively.) Features * Extremely fast generated code * Test and benchmark generation * JSON interoperability (see msgp.CopyToJSON() and msgp.UnmarshalAsJSON()) * Support for complex type declarations * Native support for Go's time.Time, complex64, and complex128 types * Generation of both byte-oriented and io.Reader/io.Writer-oriented methods * Support for arbitrary type system extensions * Preprocessor directives * File-based dependency model means fast codegen regardless of source tree size. Consider the following: const Eight = 8 type MyInt int type Data byte type Struct struct { Which map*MyInt `msg:"which"` Other Data `msg:"other"` Nums float64 `msg:"nums"` } As long as the declarations of MyInt and Data are in the same file as Struct, the parser will determine that the type information for MyInt and Data can be passed into the definition of Struct before its methods are generated. Extensions MessagePack supports defining your own types through "extensions," which are just a tuple of the data "type" (int8) and the raw binary. You can see a worked example in the wiki. Status Mostly stable, in that no breaking changes have been made to the /msgp library in more than a year. Newer versions of the code may generate different code than older versions for performance reasons. I (@philhofer) am aware of a number of stability-critical commercial applications that use this code with good results. But, caveat emptor. You can read more about how msgp maps MessagePack types onto Go types in the wiki . Here some of the known limitations/restrictions: * Identifiers from outside the processed source file are assumed (optimistically) to satisfy the generator's interfaces. If this isn't the case, your code will fail to compile. * Like most serializers, chan and func fields are ignored, as well as non-exported fields. * Encoding of interface{} is limited to built-ins or types that have explicit encoding methods. * _Maps must have string keys._ This is intentional (as it preserves JSON interop.) Although non-string map keys are not forbidden by the MessagePack standard, many serializers impose this restriction. (It also means _any_ well-formed struct can be de-serialized into a mapinterface{}.) The only exception to this rule is that the deserializers will allow you to read map keys encoded as bin types, due to the fact that some legacy encodings permitted this. (However, those values will still be cast to Go strings, and they will be converted to str types when re-encoded. It is the responsibility of the user to ensure that map keys are UTF-8 safe in this case.) The same rules hold true for JSON translation. If the output compiles, then there's a pretty good chance things are fine. (Plus, we generate tests for you.) _Please, please, please_ file an issue if you think the generator is writing broken code. Performance If you like benchmarks, see here and here . As one might expect, the generated methods that deal with byte are faster for small objects, but the io.Reader/Writer methods are generally more memory-efficient (and, at some point, faster) for large (> 2KB) objects. JAKM/MSGPACK-CLI HTTPS://GITHUB.COM/JAKM/MSGPACK-CLI MSGPACK-CLI msgpack-cli is command line tool that converts data from JSON to Msgpack and vice versa. Also allows calling RPC methods via msgpack-rpc . INSTALLATION % go get github.com/jakm/msgpack-cli Debian packages and Windows binaries are available on project's Releases page . USAGE msgpack-cli Usage: msgpack-cli encode msgpack-cli decode msgpack-cli rpc msgpack-cli -h | --help msgpack-cli --version Commands: encode Encode data from input file to STDOUT decode Decode data from input file to STDOUT rpc Call RPC method and write result to STDOUT Options: -h --help Show this help message and exit --version Show version --out= Write output data to file instead of STDOUT --file= File where parameters or RPC method are read from --pp Pretty-print - indent output JSON data --timeout= Timeout of RPC call --disable-int64-conv Disable the default behaviour such that JSON numbers are converted to float64 or int64 numbers by their meaning, all result numbers will have float64 type Arguments: File where data are read from Server hostname Server port Name of RPC method Parameters of RPC method in JSON format EXAMPLES Encoding/decoding: $ cat test.json { "firstName": "John", "lastName": "Smith", "isAlive": true, "age": 25, "height_cm": 167.6, "address": { "streetAddress": "21 2nd Street", "city": "New York", "state": "NY", "postalCode": "10021-3100" }, "phoneNumbers": , "children": , "spouse": null } $ $ msgpack-cli encode test.json --out test.bin $ $ ls -l test.* | awk '{print $9, $5}' test.bin 242 test.json 429 $ $ msgpack-cli decode test.bin --pp # pretty-print { "address": { "city": "New York", "postalCode": "10021-3100", "state": "NY", "streetAddress": "21 2nd Street" }, "age": 25, "children": , "firstName": "John", "height_cm": 167.6, "isAlive": true, "lastName": "Smith", "phoneNumbers": , "spouse": null } RPC calling: $ # zero params $ msgpack-cli rpc localhost 8000 echo $ $ # single param $ msgpack-cli rpc localhost 8000 echo 3.14159 $ $ # multiple params (as json array) $ msgpack-cli rpc localhost 8000 echo '' YMOFEN/SIMPLEMSGPACK.NET HTTPS://GITHUB.COM/YMOFEN/SIMPLEMSGPACK.NET SIMPLEMSGPACK.NET MessagePack implementation for C# / msgpack.org Binary files distributed via the NuGet package SimpleMsgPack . It's like JSON but small and fast. unit Owner: D10.Mofen contact: qq:185511468, email:ymofen@diocp.org homepage:www.diocp.org if you find any bug, please contact me! WORKS WITH .NET Framework 4.x Code Example MsgPack msgpack = new MsgPack(); msgpack.ForcePathObject("p.name").AsString = "张三"; msgpack.ForcePathObject("p.age").AsInteger = 25; msgpack.ForcePathObject("p.datas").AsArray.Add(90); msgpack.ForcePathObject("p.datas").AsArray.Add(80); msgpack.ForcePathObject("p.datas").AsArray.Add("李四"); msgpack.ForcePathObject("p.datas").AsArray.Add(3.1415926); // pack file msgpack.ForcePathObject("p.filedata").LoadFileAsBytes("C:\\a.png"); // pack msgPack binary byte packData = msgpack.Encode2Bytes(); MsgPack unpack_msgpack = new MsgPack(); // unpack msgpack unpack_msgpack.DecodeFromBytes(packData); System.Console.WriteLine("name:{0}, age:{1}", unpack_msgpack.ForcePathObject("p.name").AsString, unpack_msgpack.ForcePathObject("p.age").AsInteger); Console.WriteLine("=================================="); System.Console.WriteLine("use index property, Length{0}:{1}", unpack_msgpack.ForcePathObject("p.datas").AsArray.Length, unpack_msgpack.ForcePathObject("p.datas").AsArray.AsString ); Console.WriteLine("=================================="); Console.WriteLine("use foreach statement:"); foreach (MsgPack item in unpack_msgpack.ForcePathObject("p.datas")) { Console.WriteLine(item.AsString); } // unpack filedata unpack_msgpack.ForcePathObject("p.filedata").SaveBytesToFile("C:\\b.png"); Console.Read(); YMOFEN/THEALCHEMIST HTTPS://GITHUB.COM/YMOFEN/THEALCHEMIST THE ALCHEMIST Elixir implementation bundle for MessagePack Elixir implementation relies on mururu/msgpack-elixir Pure erlang implementation used with msgpack/msgpack-erlang USAGE ### In pure elixir: # pack TheAlchemist.pack() #=> { :ok, <<147,1,2,3>> } TheAlchemist.pack!() #=> <<147,1,2,3>> # unpack TheAlchemist.unpack(<<147,1,2,3>>) #=> { :ok, } TheAlchemist.unpack!(<<147,1,2,3>>) #=> # unpack_once TheAlchemist.unpack_once(<<147,1,2,3,4>>) #=> {:ok, {, <<4>>}} TheAlchemist.unpack_once!(<<147,1,2,3,4>>) #=> {, <<4>>} ### With underlying erlang: Deserialization msgpacked = TheAlchemistErl.pack('{"test":false}') TheAlchemistErl.unpack(msgpacked) Stream deserialization mat = TheAlchemistErl.unpack_stream(data) TheAlchemistErl.unpack_stream(elem(mat, 1)), String type Now this supports string type! opt = TheAlchemistErl.unpack(TheAlchemistErl.pack("埼玉kanji", opt), opt) => {:ok, "埼玉kanji"} Map Style Since Erlang/OTP 17.0 msgpack:pack(#{ <<"key">> => <<"value">> }, ). Or use old jiffy/jsx style msgpack:pack({}, ), msgpack:pack(, ). RUNNING INTERACTIVELY Firstly make all to ensure all source files are compiled after that you can bring the ERL shell with make run If you want to bring the IEX you can type make runex MSGPACK4Z/MSGPACK4Z-CORE HTTPS://GITHUB.COM/MSGPACK4Z/MSGPACK4Z-CORE MSGPACK4Z-CORE * https://msgpack.org/ * https://github.com/msgpack/msgpack/blob/master/spec.md example * https://github.com/msgpack4z/msgpack4z-core/blob/master/src/test/scala/msgpack4z/CaseClassExample.scala * https://github.com/xuwei-k/msgpack-json/blob/640878e88171/src/main/scala/App.scala latest stable version libraryDependencies += "com.github.xuwei-k" %% "msgpack4z-core" % "0.3.9" * sxr snapshot version libraryDependencies += "com.github.xuwei-k" %% "msgpack4z-core" % "0.3.10-SNAPSHOT" resolvers += Opts.resolver.sonatypeSnapshots * sxr dependency graph for scalaz 7.1.x https://github.com/msgpack4z/msgpack4z-core/tree/0.2.x MSGPACK4Z/MSGPACK4Z-ARGONAUT HTTPS://MSGPACK4Z.GITHUB.IO/GRAPH.SVG MSGPACK4Z-ARGONAUT * https://msgpack.org/ * https://github.com/msgpack/msgpack/blob/master/spec.md latest stable version libraryDependencies += "com.github.xuwei-k" %% "msgpack4z-argonaut" % "0.5.1" * sxr snapshot version libraryDependencies += "com.github.xuwei-k" %% "msgpack4z-argonaut" % "0.5.2-SNAPSHOT" resolvers += Opts.resolver.sonatypeSnapshots * sxr JAKM/TXMSGPACKRPC HTTPS://GITHUB.COM/JAKM/TXMSGPACKRPC TXMSGPACKRPC For the latest source code, see http://github.com/jakm/txmsgpackrpc txmsgpackrpc is a library for writing asynchronous msgpack-rpc servers and clients in Python, using Twisted framework . Library is based on txMsgpack , but some improvements and fixes were made. FEATURES * user friendly API * modular object model * working timeouts and reconnecting * connection pool support * TCP, SSL, UDP and UNIX sockets PYTHON 3 NOTE To use UNIX sockets with Python 3 please use Twisted framework 15.3.0 and above. DEPENDENCIES * msgpack-python https://pypi.python.org/pypi/msgpack-python/ * Twisted http://twistedmatrix.com/trac/ INSTALLATION % pip install txmsgpackrpc Debian packages are available on project's Releases page . TCP EXAMPLE Computation of PI using Chudnovsky algorithm in subprocess. For details, see http://www.craig-wood.com/nick/articles/pi-chudnovsky/. Results Computation of PI with 5 places finished in 0.022390 seconds Computation of PI with 100 places finished in 0.037856 seconds Computation of PI with 1000 places finished in 0.038070 seconds Computation of PI with 10000 places finished in 0.073907 seconds Computation of PI with 100000 places finished in 6.741683 seconds Computation of PI with 5 places finished in 0.001142 seconds Computation of PI with 100 places finished in 0.001182 seconds Computation of PI with 1000 places finished in 0.001206 seconds Computation of PI with 10000 places finished in 0.001230 seconds Computation of PI with 100000 places finished in 0.001255 seconds Computation of PI with 1000000 places finished in 432.574457 seconds Computation of PI with 1000000 places finished in 402.551226 seconds DONE Server from __future__ import print_function from collections import defaultdict from twisted.internet import defer, reactor, utils from twisted.python import failure from txmsgpackrpc.server import MsgpackRPCServer pi_chudovsky_bs = ''' """ Python3 program to calculate Pi using python long integers, binary splitting and the Chudnovsky algorithm See: http://www.craig-wood.com/nick/articles/pi-chudnovsky/ for more info Nick Craig-Wood """ import math from time import time def sqrt(n, one): """ Return the square root of n as a fixed point number with the one passed in. It uses a second order Newton-Raphson convgence. This doubles the number of significant figures on each iteration. """ # Use floating point arithmetic to make an initial guess floating_point_precision = 10**16 n_float = float((n * floating_point_precision) // one) / floating_point_precision x = (int(floating_point_precision * math.sqrt(n_float)) * one) // floating_point_precision n_one = n * one while 1: x_old = x x = (x + n_one // x) // 2 if x == x_old: break return x def pi_chudnovsky_bs(digits): """ Compute int(pi * 10**digits) This is done using Chudnovsky's series with binary splitting """ C = 640320 C3_OVER_24 = C**3 // 24 def bs(a, b): """ Computes the terms for binary splitting the Chudnovsky infinite series a(a) = +/- (13591409 + 545140134*a) p(a) = (6*a-5)*(2*a-1)*(6*a-1) b(a) = 1 q(a) = a*a*a*C3_OVER_24 returns P(a,b), Q(a,b) and T(a,b) """ if b - a == 1: # Directly compute P(a,a+1), Q(a,a+1) and T(a,a+1) if a == 0: Pab = Qab = 1 else: Pab = (6*a-5)*(2*a-1)*(6*a-1) Qab = a*a*a*C3_OVER_24 Tab = Pab * (13591409 + 545140134*a) # a(a) * p(a) if a & 1: Tab = -Tab else: # Recursively compute P(a,b), Q(a,b) and T(a,b) # m is the midpoint of a and b m = (a + b) // 2 # Recursively calculate P(a,m), Q(a,m) and T(a,m) Pam, Qam, Tam = bs(a, m) # Recursively calculate P(m,b), Q(m,b) and T(m,b) Pmb, Qmb, Tmb = bs(m, b) # Now combine Pab = Pam * Pmb Qab = Qam * Qmb Tab = Qmb * Tam + Pam * Tmb return Pab, Qab, Tab # how many terms to compute DIGITS_PER_TERM = math.log10(C3_OVER_24/6/2/6) N = int(digits/DIGITS_PER_TERM + 1) # Calclate P(0,N) and Q(0,N) P, Q, T = bs(0, N) one = 10**digits sqrtC = sqrt(10005*one, one) return (Q*426880*sqrtC) // T if __name__ == "__main__": import sys digits = int(sys.argv) pi = pi_chudnovsky_bs(digits) print(pi) ''' def set_timeout(deferred, timeout=30): def callback(value): if not watchdog.called: watchdog.cancel() return value deferred.addBoth(callback) watchdog = reactor.callLater(timeout, defer.timeout, deferred) class ComputePI(MsgpackRPCServer): def __init__(self): self.waiting = defaultdict(list) self.results = {} def remote_PI(self, digits, timeout=None): if digits in self.results: return defer.succeed(self.results) d = defer.Deferred() if digits not in self.waiting: subprocessDeferred = self.computePI(digits, timeout) def callWaiting(res): waiting = self.waiting del self.waiting if isinstance(res, failure.Failure): func = lambda d: d.errback(res) else: func = lambda d: d.callback(res) for d in waiting: func(d) subprocessDeferred.addBoth(callWaiting) self.waiting.append(d) return d def computePI(self, digits, timeout): d = utils.getProcessOutputAndValue('/usr/bin/python', args=('-c', pi_chudovsky_bs, str(digits))) def callback((out, err, code)): if code == 0: pi = int(out) self.results = pi return pi else: return failure.Failure(RuntimeError('Computation failed: ' + err)) if timeout is not None: set_timeout(d, timeout) d.addCallback(callback) return d def main(): server = ComputePI() reactor.listenTCP(8000, server.getStreamFactory()) if __name__ == '__main__': reactor.callWhenRunning(main) reactor.run() Client from __future__ import print_function import sys import time from twisted.internet import defer, reactor, task from twisted.python import failure @defer.inlineCallbacks def main(): try: from txmsgpackrpc.client import connect c = yield connect('localhost', 8000, waitTimeout=900) def callback(res, digits, start_time): if isinstance(res, failure.Failure): print('Computation of PI with %d places failed: %s' % (digits, res.getErrorMessage()), end='\n\n') else: print('Computation of PI with %d places finished in %f seconds' % (digits, time.time() - start_time), end='\n\n') sys.stdout.flush() defers = for _ in range(2): for digits in (5, 100, 1000, 10000, 100000, 1000000): d = c.createRequest('PI', digits, 600) d.addBoth(callback, digits, time.time()) defers.append(d) # wait for 30 seconds yield task.deferLater(reactor, 30, lambda: None) yield defer.DeferredList(defers) print('DONE') except Exception: import traceback traceback.print_exc() finally: reactor.stop() if __name__ == '__main__': reactor.callWhenRunning(main) reactor.run() MULTICAST UDP EXAMPLE Example servers join to group 224.0.0.5 and listen on port 8000. Their only method echo returns its parameter. Client joins group to 224.0.0.5, sends multicast request to group on port 8000 and waits for 5 seconds for responses. If some responses are received, protocol callbacks with tuple of results and individual parts are checked for errors. If no responses are received, protocol errbacks with TimeoutError. Because there is no common way to determine number of peers in group, MsgpackMulticastDatagramProtocol always wait for responses until waitTimeout expires. $ # setup multicast routing $ ip route add 224.0.0.0/4 dev eth0 $ echo 1 > /proc/sys/net/ipv4/ip_forward $ $ # start servers listening on port 8000 $ python examples/tx_rpc_server_udp_multicast.py & 3584 $ python examples/tx_rpc_server_udp_multicast.py & 3585 $ python examples/tx_rpc_server_udp_multicast.py & 3586 $ python examples/tx_rpc_server_udp_multicast.py & 3587 $ python examples/tx_rpc_server_udp_multicast.py & 3588 $ $ # execute client $ python examples/tx_rpc_client_udp_multicast.py Received results from 5 peers $ Server from twisted.internet import defer, reactor, task from txmsgpackrpc.server import MsgpackRPCServer class EchoRPC(MsgpackRPCServer): @defer.inlineCallbacks def remote_echo(self, value, delay=None, msgid=None): if delay is not None: yield task.deferLater(reactor, delay, lambda: None) defer.returnValue(value) def main(): server = EchoRPC() reactor.listenMulticast(8000, server.getMulticastProtocol('228.0.0.5', ttl=5), listenMultiple=True) if __name__ == '__main__': reactor.callWhenRunning(main) reactor.run() Client from __future__ import print_function from twisted.internet import defer, reactor @defer.inlineCallbacks def main(): try: from txmsgpackrpc.client import connect_multicast c = yield connect_multicast('228.0.0.5', 8000, ttl=5, waitTimeout=5) data = { 'firstName': 'John', 'lastName': 'Smith', 'isAlive': True, 'age': 25, 'height_cm': 167.6, 'address': { 'streetAddress': "21 2nd Street", "city": 'New York', "state": 'NY', 'postalCode': '10021-3100' }, 'phoneNumbers': , 'children': , 'spouse': None } results = yield c.createRequest('echo', data) assert isinstance(results, tuple) print('Received results from %d peers' % len(results)) for i, result in enumerate(results): if result != data: print('Result %d mismatch' % i) print(result) except Exception: import traceback traceback.print_exc() finally: reactor.stop() if __name__ == '__main__': reactor.callWhenRunning(main) reactor.run() JONGHOUGH/MSGPACK-J HTTPS://GITHUB.COM/JONGHOUGH/MSGPACK-J MSGPACK-J Implementation of the Message Pack serialization format in J. ###Usage import into projects and open in jqt or other. It's very simple. ###Examples pack packs J types into a byte string. unpack reverses this. `pack 'hello world'` `�hello world` `unpack �hello world` `'hello world'` PACKOBJ packs J types into a hex string (string literal). UNPACKOBJ reverses this. e.g. `packObj 'Hello World'` `ab48656c6c6f20576f726c64` `unpackObj 'ab48656c6c6f20576f726c64'` `Hello World` ###More Usage Example: `pack 2;4.67;'hello, world'` `���@┐�┼z�G��hello, world` `packObj 2;4.67;'hello, world'` `9302cb4012ae147ae147aeac68656c6c6f2c20776f726c64` JSON representation: Example: `unpackObj '81a46461746183a2696401a673636f72657394cb400999999999999acb4016cccccccccccdcb40091eb851eb851fcb4007333333333333a56f7468657283a4736f6d65d1f2b8a46d6f7265ccc8a4646174610c'` returns a nested dictionary. JSON representation: `{"data":{"id":1,"scores":,"other":{"some":_3400,"more":200,"data":12}}}` ###Handling dictionary / hashmap datatypes Since J has no native Dictionary / Hashmap type, one has been implemented for the purposes of MsgPack serialization. Construction: `HM =: '' conew 'HashMap'` This will instantiate a new HashMap object. `set__HM 'key';'value'` This will add a key value pair to the dicitonary. Note the length of the boxed array argument must be two. i.e. if the value is an array itself, then it must be boxed together before appending to the key value. `get__HM 'key'` This will return the value for the given key, if one exists. To pack a HashMap: `packObj s: HM` Here HM is the HashMap reference name. It must be symbolized first, before packing. Furthermore, to add a HashMap as a value of another HashMap: `set__HM 'hashmapkey';s:HM2` The inner HashMap reference (HM2) must be symbolized before adding to the dictionary. If you are adding a list of HashMaps to the parent HashMap: `set__HM 'key'; <(s:HM2;s:HM3;s:HM4)` Note the HashMap array is boxed so that the argument for set is of length two. Since the HashMap HM stores the reference to the child HashMaps as symbols, they must be desymbolized if retrieved. e.g. `ChildHM =: getHashmapFromValue_HashMap_ get__HM 'mychildHashMapkey'` Here, GETHASHMAPFROMVALUE_HASHMAP_ ensures that the retrieved object is a reference to a hashmap, as is wanted. When unpacking data, assuming the root object is a dictionary / hashmap: `HM =: 5 s: unpackObj 'some serialized data'` 5 S: must be called to desymbolize the reference to the HashMap. Furthermore, all child HashMaps of HM must also be desymbolized too. ` A2/MESSAGEPACK.SWIFT HTTPS://GITHUB.COM/A2/MESSAGEPACK.SWIFT MESSAGEPACK.SWIFT A fast, zero-dependency MessagePack implementation written in Swift 4. Supports Apple platforms and Linux. INSTALLATION CocoaPods To use CocoaPods, add the following to your Podfile: pod 'MessagePack.swift', '~> 3.0' Carthage To use Carthage, add the following to your Cartfile: github "a2/MessagePack.swift" ~> 3.0 SPM (Swift Package Manager) You can easily integrate MessagePack.swift in your app with SPM. Just add MessagePack.swift as a dependency: import PackageDescription let package = Package( name: "MyAwesomeApp", dependencies: ) VERSION * Versions 3.x support Swift 4. * Support for Swift 3 was dropped after 2.1.1 . * Support for Swift 2 was dropped after 1.2.0 . AUTHORS Alexsander Akers, me@a2.io LICENSE MessagePack.swift is available under the MIT license. See the LICENSE file for more info. AKIRADEVELOPER/MSGPACK-NIM HTTPS://GITHUB.COM/AKIRADEVELOPER/MSGPACK-NIM I will start this project once Nim compiler reaches 1.0 MSGPACK-NIM A MessagePack binding for Nim API: https://rawgit.com/akiradeveloper/msgpack-nim/master/msgpack.html msgpack-nim currently provides only the basic functionality. Please see what's listed in Todo section. Compared to other language bindings, it's well-tested by 1000 auto-generated test cases by Haskell QuickCheck, which always runs on every commit to Github repository. Please try make quickcheck on your local machine to see what happens (It will take a bit while. Be patient). Have a nice packing! INSTALL $ nimble update $ nimble install msgpack EXAMPLE import msgpack import streams # You can use any stream subclasses to serialize/deserialize # messages. e.g. FileStream let st: Stream = newStringStream() assert(st.getPosition == 0) # Type checking protects you from making trivial mistakes. # Now we pack {"a":, "b":} but more complex # combination of any Msg types is allowed. # # In xs we can mix specific conversion (PFixNum) and generic # conversion (unwrap). let xs: Msg = wrap(@) let ys: Msg = wrap(@) st.pack(ys.wrap) # Serialize! # We need to reset the cursor to the beginning of the target # byte sequence. st.setPosition(0) let msg = st.unpack # Deserialize! # output: # a # 5 # -3 # b # 1 # 2 # 3 for e in msg.unwrapMap: echo e.key.unwrapStr for e in e.val.unwrapArray: echo e.unwrapInt TODO * Implement unwrapInto to convert Msg object to Nim object handily * Evaluate performance and scalability * Talk with offical Ruby implementation * Don't repeat yourself: The code now has too much duplications. Using templates? AUTHOR Akira Hayakawa (ruby.wktk@gmail.com) LUDOCODE/MPACK HTTPS://GITHUB.COM/LUDOCODE/MPACK INTRODUCTION MPack is a C implementation of an encoder and decoder for the MessagePack serialization format. It is: * Simple and easy to use * Secure against untrusted data * Lightweight, suitable for embedded * Extensively documented * Extremely fast The core of MPack contains a buffered reader and writer, and a tree-style parser that decodes into a tree of dynamically typed nodes. Helper functions can be enabled to read values of expected type, to work with files, to allocate strings automatically, to check UTF-8 encoding, and more. The MPack code is small enough to be embedded directly into your codebase. Simply download the amalgamation package and add mpack.h and mpack.c to your project. The MPack featureset can be customized at compile-time to set which features, components and debug checks are compiled, and what dependencies are available. BUILD STATUS MPack is beta software under development. TRAVIS-CI APPVEYOR COVERALLS.IO THE NODE API The Node API parses a chunk of MessagePack data into an immutable tree of dynamically-typed nodes. A series of helper functions can be used to extract data of specific types from each node. // parse a file into a node tree mpack_tree_t tree; mpack_tree_init_filename(&tree, "homepage-example.mp", 0); mpack_tree_parse(&tree); mpack_node_t root = mpack_tree_root(&tree); // extract the example data on the msgpack homepage bool compact = mpack_node_bool(mpack_node_map_cstr(root, "compact")); int schema = mpack_node_i32(mpack_node_map_cstr(root, "schema")); // clean up and check for errors if (mpack_tree_destroy(&tree) != mpack_ok) { fprintf(stderr, "An error occurred decoding the data!\n"); return; } Note that no additional error handling is needed in the above code. If the file is missing or corrupt, if map keys are missing or if nodes are not in the expected types, special "nil" nodes and false/zero values are returned and the tree is placed in an error state. An error check is only needed before using the data. The above example decodes into allocated pages of nodes. A fixed node pool can be provided to the parser instead in memory-constrained environments. For maximum performance and minimal memory usage, the Expect API can be used to parse data of a predefined schema. THE WRITE API The Write API encodes structured data to MessagePack. // encode to memory buffer char* data; size_t size; mpack_writer_t writer; mpack_writer_init_growable(&writer, &data, &size); // write the example on the msgpack homepage mpack_start_map(&writer, 2); mpack_write_cstr(&writer, "compact"); mpack_write_bool(&writer, true); mpack_write_cstr(&writer, "schema"); mpack_write_uint(&writer, 0); mpack_finish_map(&writer); // finish writing if (mpack_writer_destroy(&writer) != mpack_ok) { fprintf(stderr, "An error occurred encoding the data!\n"); return; } // use the data do_something_with_data(data, size); free(data); In the above example, we encode to a growable memory buffer. The writer can instead write to a pre-allocated or stack-allocated buffer, avoiding the need for memory allocation. The writer can also be provided with a flush function (such as a file or socket write function) to call when the buffer is full or when writing is done. If any error occurs, the writer is placed in an error state. The writer will flag an error if too much data is written, if the wrong number of elements are written, if the data could not be flushed, etc. No additional error handling is needed in the above code; any subsequent writes are ignored when the writer is in an error state, so you don't need to check every write for errors. Note in particular that in debug mode, the mpack_finish_map() call above ensures that two key/value pairs were actually written as claimed, something that other MessagePack C/C++ libraries may not do. COMPARISON WITH OTHER PARSERS MPack is rich in features while maintaining very high performance and a small code footprint. Here's a short feature table comparing it to other C parsers: MPack (v0.8) msgpack-c (v1.3.0) CMP (v14) No libc requirement ✓ ✓ Growable memory writer ✓ ✓ File I/O helpers ✓ ✓ Tree parser ✓ ✓ Propagating errors ✓ ✓ Compound size tracking ✓ Incremental parser ✓ ✓ Incremental range/match helpers ✓ Tree stream parser ✓ UTF-8 verification ✓ A larger feature comparison table is available here which includes descriptions of the various entries in the table. This benchmarking suite compares the performance of MPack to other implementations of schemaless serialization formats. MPack outperforms all JSON and MessagePack libraries, and in some tests MPack is several times faster than RapidJSON for equivalent data. WHY NOT JUST USE JSON? Conceptually, MessagePack stores data similarly to JSON: they are both composed of simple values such as numbers and strings, stored hierarchically in maps and arrays. So why not just use JSON instead? The main reason is that JSON is designed to be human-readable, so it is not as efficient as a binary serialization format: * Compound types such as strings, maps and arrays are delimited, so appropriate storage cannot be allocated upfront. The whole object must be parsed to determine its size. * Strings are not stored in their native encoding. Special characters such as quotes and backslashes must be escaped when written and converted back when read. * Numbers are particularly inefficient (especially when parsing back floats), making JSON inappropriate as a base format for structured data that contains lots of numbers. * Binary data is not supported by JSON at all. Small binary blobs such as icons and thumbnails need to be Base64 encoded or passed out-of-band. The above issues greatly increase the complexity of the decoder. Full-featured JSON decoders are quite large, and minimal decoders tend to leave out such features as string unescaping and float parsing, instead leaving these up to the user or platform. This can lead to hard-to-find platform-specific and locale-specific bugs, as well as a greater potential for security vulnerabilites. This also significantly decreases performance, making JSON unattractive for use in applications such as mobile games. While the space inefficiencies of JSON can be partially mitigated through minification and compression, the performance inefficiencies cannot. More importantly, if you are minifying and compressing the data, then why use a human-readable format in the first place? RUNNING THE UNIT TESTS The MPack build process does not build MPack into a library; it is used to build and run the unit tests. You do not need to build MPack or the unit testing suite to use MPack. On Linux, the test suite uses SCons and requires Valgrind, and can be run in the repository or in the amalgamation package. Run scons to build and run the test suite in full debug configuration. On Windows, there is a Visual Studio solution, and on OS X, there is an Xcode project for building and running the test suite. You can also build and run the test suite in all supported configurations, which is what the continuous integration server will build and run. If you are on 64-bit, you will need support for cross-compiling to 32-bit, and running 32-bit binaries with 64-bit Valgrind. On Ubuntu, you'll need libc6-dbg:i386. On Arch you'll need gcc-multilib or lib32-clang, and valgrind-multilib. Use scons all=1 -j16 (or some appropriate thread count) to build and run all tests. 3HREN/MSGPACK-RUST HTTPS://GITHUB.COM/3HREN/MSGPACK-RUST RMP - RUST MESSAGEPACK RMP is a pure Rust MessagePack implementation. This repository consists of three separate crates: the RMP core and two implementations to ease serializing and deserializing Rust structs. CRATES.IO API DOCUMENTATION RMP RMP RMP-SERIALIZE RMP Serialize RMPS RMP Serde RMPV RMP Value FEATURES * CONVENIENT API RMP is designed to be lightweight and straightforward. There are low-level API, which gives you full control on data encoding/decoding process and makes no heap allocations. On the other hand there are high-level API, which provides you convenient interface using Rust standard library and compiler reflection, allowing to encode/decode structures using derive attribute. * ZERO-COPY VALUE DECODING RMP allows to decode bytes from a buffer in a zero-copy manner easily and blazingly fast, while Rust static checks guarantees that the data will be valid as long as the buffer lives. * CLEAR ERROR HANDLING RMP's error system guarantees that you never receive an error enum with unreachable variant. * ROBUST AND TESTED This project is developed using TDD and CI, so any found bugs will be fixed without breaking existing functionality. REQUIREMENTS * Rust 1.16 VERSIONING This project adheres to Semantic Versioning . However until 1.0.0 comes there will be the following rules: * Any API/ABI breaking changes will be notified in the changelog explicitly and results in minor version bumping. * API extending features results in patch version bumping. * Non-breaking bug fixes and performance improving results in patch version bumping. JANGKO/MSGPACK4NIM HTTPS://GITHUB.COM/JANGKO/MSGPACK4NIM MSGPACK4NIM MessagePack implementation written in pure nim why another implementation? I am fully aware of another msgpack implementation written in nim. But I want something easier to use. Another motivation come from the nim language itself. The current version of nim compiler offer many improvements, including 'generics ' specialization. I found out nim compiler is smart enough to make serialization/deserialization to/from msgpack easy and convenient. REQUIREMENT: nim ver 0.11.2 or later EXAMPLE import msgpack4nim, streams type #lets try with a rather complex object CustomType = object count: int content: seq name: string ratio: float attr: array ok: bool proc initCustomType(): CustomType = result.count = -1 result.content = @ result.name = "custom" result.ratio = 1.0 for i in 0..5: result.attr = i result.ok = false var x = initCustomType() #you can use another stream compatible #class here e.g. FileStream var s = newStringStream() s.pack(x) #here the magic happened s.setPosition(0) var xx: CustomType s.unpack(xx) #and here too assert xx == x echo "OK ", xx.name see? you only need to call 'pack' and 'unpack', and the compiler do the hard work for you. Very easy, convenient, and works well if you think setting up a StringStream too much for you, you can simply call pack(yourobject) and it will return a string containing msgpack data. var a = @ var buf = pack(a) var aa: seq unpack(buf, aa) assert a == aa in case the compiler cannot decide how to serialize or deserialize your very very complex object, you can help it in easy way by defining your own handler pack_type/unpack_type type #not really complex, just for example mycomplexobject = object a: someSimpleType b: someSimpleType #help the compiler to decide proc pack_type*(s: Stream, x: mycomplexobject) = s.pack(x.a) # let the compiler decide s.pack(x.b) # let the compiler decide #help the compiler to decide proc unpack_type*(s: Stream, x: var complexobject) = s.unpack(x.a) s.unpack(x.b) var s: newStringStream() var x: mycomplexobject s.pack(x) #pack as usual s.setPosition(0) s.unpack(x) #unpack as usual DATA CONVERSION NIM MSGPACK int8/16/32/64 int8/16/32/64 uint8/16/32/64 uint8/16/32/64 true/false/nil true/false/nil procedural type ignored cstring ignored pointer ignored ptr see ref-types ref see ref-types circular ref see ref-types distinct types converted to base type float32/64 float32/64 string string8/16/32 array/seq array set array range/subrange int8/16/32/64 enum int8/16/32/64 IntSet,Doubly/SinglyLinkedList array Doubly/SinglyLinkedRing array Queue,HashSet,OrderedSet array Table,TableRef map OrderedTable,OrderedTableRef map StringTableRef map CritBitTree map CritBitTree array object/tuple array/map OBJECT AND TUPLE object and tuple by default converted to msgpack array, however you can tell the compiler to convert it to map by supplying --define:msgpack_obj_to_map nim c --define:msgpack_obj_to_map yourfile.nim or --define:msgpack_obj_to_stream to convert object/tuple fields _value_ into stream of msgpack objects nim c --define:msgpack_obj_to_stream yourfile.nim What this means? It means by default, each object/tuple will be converted to one msgpack array contains field(s) value only without their field(s) name. If you specify that the object/tuple will be converted to msgpack map, then each object/tuple will be converted to one msgpack map contains key-value pairs. The key will be field name, and the value will be field value. If you specify that the object/tuple will be converted to msgpack stream, then each object/tuple will be converted into one or more msgpack's type for each object's field and then the resulted stream will be concatenated to the msgpack stream buffer. Which one should I use? Usually, other msgpack libraries out there convert object/tuple/record/struct or whatever structured data supported by the language into msgpack array, but always make sure to consult the documentation first. If both of the serializer and deserializer agreed to one convention, then usually there will be no problem. No matter which library/language you use, you can exchange msgpack data among them. REF-TYPES: _ref something_ : * if ref value is nil, it will be packed into msgpack nil, and when unpacked, you will get nil too * if ref value not nil, it will be dereferenced e.g. pack(val) or unpack(val) * ref subject to some restriction. see RESTRICTION below * ptr will be treated like ref during pack * unpacking ptr will invoke alloc, so you must dealloc it _circular reference_: altough detecting circular reference is not too difficult(using set of pointers), the current implementation does not provide circular reference detection. If you pack something contains circular reference, you know something bad will happened RESTRICTION: For objects their type is NOT serialized. This means essentially that it does not work if the object has some other runtime type than its compiletime type: import streams, msgpack4nim type TA = object of RootObj TB = object of TA f: int var a: ref TA b: ref TB new(b) a = b echo stringify(pack(a)) #produces "" or "{ }" #not "" or '{ "f" : 0 }' LIMITATION: these types will be ignored: * procedural type * cstring(it is not safe to assume it always terminated by null) * pointer these types cannot be automatically pack/unpacked: * _void_ (will cause compile time error) however, you can provide your own handler for cstring and pointer GOTCHAS: because data conversion did not preserve original data types, the following code is perfectly valid and will raise no exception import msgpack4nim, streams, tables, sets, strtabs type Horse = object legs: int foals: seq attr: Table Cat = object legs: uint8 kittens: HashSet traits: StringTableRef proc initHorse(): Horse = result.legs = 4 result.foals = @ result.attr = initTable() result.attr = "black" result.attr = "120mph" var stallion = initHorse() var tom: Cat var buf = pack(stallion) #pack a Horse here unpack(buf, tom) #abracadabra, it will unpack into a Cat echo "legs: ", $tom.legs echo "kittens: ", $tom.kittens echo "traits: ", $tom.traits another gotcha: type KAB = object of RootObj aaa: int bbb: int KCD = object of KAB ccc: int ddd: int KEF = object of KCD eee: int fff: int var kk = KEF() echo stringify(pack(kk)) # will produce "{ "eee" : 0, "fff" : 0, "ccc" : 0, "ddd" : 0, "aaa" : 0, "bbb" : 0 }" # not "{ "aaa" : 0, "bbb" : 0, "ccc" : 0, "ddd" : 0, "eee" : 0, "fff" : 0 }" BIN AND EXT FORMAT this implementation provide function to encode/decode msgpack bin/ext format header, but for the body, you must write it yourself to the StringStream * proc pack_bin*(s: Stream, len: int) * proc pack_ext*(s: Stream, len: int, exttype: int8) * proc unpack_bin*(s: Stream): int * proc unpack_ext*(s: Stream): tuple import streams, msgpack4nim const exttype0 = 0 var s = newStringStream() var body = "this is the body" s.pack_ext(body.len, exttype0) s.write(body) #the same goes to bin format s.pack_bin(body.len) s.write(body) s.setPosition(0) #unpack_ext return tuple let (extype, extlen) = s.unpack_ext() var extbody = s.readStr(extlen) assert extbody == body let binlen = s.unpack_bin() var binbody = s.readStr(binlen) assert binbody == body STRINGIFY you can convert msgpack data to readable string using stringify function type Horse = object legs: int speed: int color: string name: string var cc = Horse(legs:4, speed:150, color:"black", name:"stallion") var zz = pack(cc) echo stringify(zz) the result will be: default: msgpack_obj_to_map defined: { "legs" : 4, "speed" : 150, "color" : "black", "name" : "stallion" } msgpack_obj_to_stream defined: 4 150 "black" "stallion" TOANY TOANY takes a string of msgpack data or a stream, then it will produce MSGANY which you can interrogate of it's type and value during runtime by accessing it's member MSGTYPE TOANY recognize all valid msgpack message and translate it into a group of types: msgMap, msgArray, msgString, msgBool, msgBin, msgExt, msgFloat32, msgFloat64, msgInt, msgUint, msgNull for example, MSG is a _msgpack_ data with content , you can interrogate it like this: var a = msg.toAny() assert a.msgType == msgArray assert a.arrayVal.msgType == msgInt assert a.arrayVal.intVal == 1 assert a.arrayVal.msgType == msgString assert a.arrayVal.stringVal == "hello" assert a.arrayVal.msgType == msgMap assert a.arrayVal.mapVal.key.msgType == msgString assert a.arrayVal.mapVal.key.stringVal == "a" assert a.arrayVal.mapVal.val.msgType == msgString assert a.arrayVal.mapVal.val.stringVal == "b" enjoy it, happy nim-ing CRYSTAL-COMMUNITY/MSGPACK-CRYSTAL HTTPS://GITHUB.COM/CRYSTAL-COMMUNITY/MSGPACK-CRYSTAL MESSAGEPACK MessagePack implementation in Crystal. INSTALLATION Add this to your application's shard.yml: dependencies: msgpack: github: crystal-community/msgpack-crystal USAGE require "msgpack" class Location include MessagePack::Serializable property lat : Float64 property lng : Float64 end class House include MessagePack::Serializable property address : String property location : Location? end house = House.from_msgpack({address: "Road12", location: {lat: 12.3, lng: 34.5}}.to_msgpack) p house # => > p house.to_msgpack # => Bytes[130, 167, 97, 100, 100, 114, 101, 115, 115, 166, 82, 111, 97, 100, ... house.address = "Something" house = House.from_msgpack(house.to_msgpack) p house # => #> house = House.from_msgpack({"address" => "Crystal Road 1234"}.to_msgpack) p house # => MORE EXAMPLES examples MSGPACK-RPC implemented by simple_rpc shard COPYRIGHT Copyright 2015 Benoist Claassen _Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License You may obtain a copy of the License at_ http://www.apache.org/licenses/LICENSE-2.0 _Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License._ ASMOD4N/MRUBY-SIMPLEMSGPACK HTTPS://GITHUB.COM/ASMOD4N/MRUBY-SIMPLEMSGPACK MRUBY-SIMPLEMSGPACK INSTALLATION First get a working copy of mruby then add conf.gem mgem: 'mruby-simplemsgpack' to the build_conf.rb of the mruby directory mruby-simplemsgpack searches for msgpack-c on your system, if it can find it it links against it, there is also a bundled version of msgpack-c included if you don't have it installed in your system. You need at least msgpack-c 1. EXAMPLE Objects can be packed with Object#to_msgpack or MessagePack.pack: packed_hash = { a: 'hash', with: }.to_msgpack packed_string = MessagePack.pack('bye') packed_hash # => "\x82\xA1a\xA4hash\xA4with\x93\x01\xA8embedded\xA5array" packed_string # => "\xA3bye" They are unpacked with MessagePack.unpack: MessagePack.unpack(packed_hash) # => { a: 'hash', with: } MessagePack.unpack(packed_string) # => 'bye' A string with multiple packed values can be unpacked by handing a block to MessagePack.unpack: packed = packed_string + packed_hash unpacked = MessagePack.unpack(packed) do |result| unpacked << result end unpacked # => When using MessagePack.unpack with a block and passing it a incomplete packed Message it returns the number of bytes it was able to unpack, if it was able to unpack the howl Message it returns self. This is helpful if the given data contains an incomplete last object and we want to continue unpacking after we have more data. packed = packed_string + packed_hash.slice(0, packed_hash.length/2) unpacked = unpacked_length = MessagePack.unpack(packed) do |result| unpacked << result end unpacked_length # => 4 (length of packed_string) unpacked # => EXTENSION TYPES To customize how objects are packed, define an extension type . By default, MessagePack packs symbols as strings and does not convert them back when unpacking them. Symbols can be preserved by registering an extension type for them: sym_ext_type = 0 MessagePack.register_pack_type(sym_ext_type, Symbol) { |symbol| symbol.to_s } MessagePack.register_unpack_type(sym_ext_type) { |data| data.to_sym } MessagePack.unpack(:symbol.to_msgpack) # => :symbol Other objects like classes can also be preserved: cls_ext_type = 1 MessagePack.register_pack_type(cls_ext_type, Class) { |cls| cls.to_s } MessagePack.register_unpack_type(cls_ext_type) { |data| data.constantize } MessagePack.unpack(Object.to_msgpack) # => Object For nil, true, false, Fixnum, Float, String, Array and Hash a registered ext type is ignored. They are always packed according to the MessagePack specification . PROC, BLOCKS OR LAMBAS If you want to pack and unpack mruby blocks take a look at the mruby-proc-irep-ext gem, it can be registered like the other extension types OVERRIDING TO_MSGPACK It's not supported to override to_msgpack, MessagePack.pack ignores it, same when that object is included in a Hash or Array. This gem treats objects like ruby does, if you want to change the way your custom Class gets handled you can add to_hash, to_ary, to_int or to_str methods so it will be packed like a Hash, Array, Fixnum or String (in that order) then. ACKNOWLEDGEMENTS This is using code from https://github.com/msgpack/msgpack-c Copyright (C) 2008-2015 FURUHASHI Sadayuki Distributed under the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt ) KAWANET/MSGPACK-LITE HTTPS://WWW.NPMJS.COM/PACKAGE/MSGPACK-LITE MSGPACK-LITE Fast Pure JavaScript MessagePack Encoder and Decoder Online demo: http://kawanet.github.io/msgpack-lite/ Features * Pure JavaScript only (No node-gyp nor gcc required) * Faster than any other pure JavaScript libraries on node.js v4 * Even faster than node-gyp C++ based msgpack library (90% FASTER on encoding) * Streaming encoding and decoding interface is also available. It's more faster. * Ready for Web browsers including Chrome, Firefox, Safari and even IE8 * Tested on Node.js v0.10, v0.12, v4, v5 and v6 as well as Web browsers Encoding and Decoding MessagePack var msgpack = require("msgpack-lite"); // encode from JS Object to MessagePack (Buffer) var buffer = msgpack.encode({"foo": "bar"}); // decode from MessagePack (Buffer) to JS Object var data = msgpack.decode(buffer); // => {"foo": "bar"} // if encode/decode receives an invalid argument an error is thrown Writing to MessagePack Stream var fs = require("fs"); var msgpack = require("msgpack-lite"); var writeStream = fs.createWriteStream("test.msp"); var encodeStream = msgpack.createEncodeStream(); encodeStream.pipe(writeStream); // send multiple objects to stream encodeStream.write({foo: "bar"}); encodeStream.write({baz: "qux"}); // call this once you're done writing to the stream. encodeStream.end(); Reading from MessagePack Stream var fs = require("fs"); var msgpack = require("msgpack-lite"); var readStream = fs.createReadStream("test.msp"); var decodeStream = msgpack.createDecodeStream(); // show multiple objects decoded from stream readStream.pipe(decodeStream).on("data", console.warn); Decoding MessagePack Bytes Array var msgpack = require("msgpack-lite"); // decode() accepts Buffer instance per default msgpack.decode(Buffer()); // decode() also accepts Array instance msgpack.decode(); // decode() accepts raw Uint8Array instance as well msgpack.decode(new Uint8Array()); Command Line Interface A CLI tool bin/msgpack converts data stream from JSON to MessagePack and vice versa. $ echo '{"foo": "bar"}' | ./bin/msgpack -Jm | od -tx1 0000000 81 a3 66 6f 6f a3 62 61 72 $ echo '{"foo": "bar"}' | ./bin/msgpack -Jm | ./bin/msgpack -Mj {"foo":"bar"} Installation $ npm install --save msgpack-lite Tests Run tests on node.js: $ make test Run tests on browsers: $ make test-browser-local open the following url in a browser: http://localhost:4000/__zuul Browser Build Browser version msgpack.min.js is also available. 50KB minified, 14KB gziped. time. Step #2: add browser property on package.json in your project. This refers the global msgpack object instead of including whole of msgpack-lite source code. { "dependencies": { "msgpack-lite": "*" }, "browser": { "msgpack-lite": "msgpack-lite/global" } } Step #3: compile it with browserify and uglifyjs . browserify src/main.js -o tmp/main.browserify.js -s main uglifyjs tmp/main.browserify.js -m -c -o js/main.min.js cp node_modules/msgpack-lite/dist/msgpack.min.js js/msgpack.min.js Step #4: load msgpack.min.js before your code. Interoperability It is tested to have basic compatibility with other Node.js MessagePack modules below: * https://www.npmjs.com/package/msgpack (1.0.2) * https://www.npmjs.com/package/msgpack-js (0.3.0) * https://www.npmjs.com/package/msgpack-js-v5 (0.3.0-v5) * https://www.npmjs.com/package/msgpack-unpack (2.1.1) * https://github.com/msgpack/msgpack-javascript (msgpack.codec) * https://www.npmjs.com/package/msgpack5 (3.3.0) * https://www.npmjs.com/package/notepack (0.0.2) Benchmarks A benchmark tool lib/benchmark.js is available to compare encoding/decoding speed (operation per second) with other MessagePack modules. It counts operations of 1KB JSON document in 10 seconds. $ npm install msgpack msgpack-js msgpack-js-v5 msgpack-unpack msgpack5 notepack $ npm run benchmark 10 OPERATION OP MS OP/S buf = Buffer(JSON.stringify(obj)); 1055200 10000 105520 obj = JSON.parse(buf); 863800 10000 86380 buf = require("msgpack-lite").encode(obj); 969100 10000 96910 obj = require("msgpack-lite").decode(buf); 600300 10000 60030 buf = require("msgpack").pack(obj); 503500 10001 50344 obj = require("msgpack").unpack(buf); 560200 10001 56014 buf = Buffer(require("msgpack.codec").msgpack.pack(obj)); 653500 10000 65349 obj = require("msgpack.codec").msgpack.unpack(buf); 367500 10001 36746 buf = require("msgpack-js-v5").encode(obj); 189500 10002 18946 obj = require("msgpack-js-v5").decode(buf); 408900 10000 40890 buf = require("msgpack-js").encode(obj); 189200 10000 18920 obj = require("msgpack-js").decode(buf); 375600 10002 37552 buf = require("msgpack5")().encode(obj); 110500 10009 11040 obj = require("msgpack5")().decode(buf); 165500 10000 16550 buf = require("notepack")().encode(obj); 847800 10000 84780 obj = require("notepack")().decode(buf); 599800 10000 59980 obj = require("msgpack-unpack").decode(buf); 48100 10002 4809 Streaming benchmark tool lib/benchmark-stream.js is also available. It counts milliseconds for 1,000,000 operations of 30 bytes fluentd msgpack fragment. This shows streaming encoding and decoding are super faster. $ npm run benchmark-stream 2 OPERATION (1000000 X 2) OP MS OP/S stream.write(msgpack.encode(obj)); 1000000 3027 330360 stream.write(notepack.encode(obj)); 1000000 2012 497017 msgpack.Encoder().on("data",ondata).encode(obj); 1000000 2956 338294 msgpack.createEncodeStream().write(obj); 1000000 1888 529661 stream.write(msgpack.decode(buf)); 1000000 2020 495049 stream.write(notepack.decode(buf)); 1000000 1794 557413 msgpack.Decoder().on("data",ondata).decode(buf); 1000000 2744 364431 msgpack.createDecodeStream().write(buf); 1000000 1341 745712 Test environment: msgpack-lite 0.1.14, Node v4.2.3, Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz MessagePack Mapping Table The following table shows how JavaScript objects (value) will be mapped to MessagePack formats and vice versa. SOURCE VALUE MESSAGEPACK FORMAT VALUE DECODED null, undefined nil format family null Boolean (true, false) bool format family Boolean (true, false) Number (32bit int) int format family Number (int or double) Number (64bit double) float format family Number (double) String str format family String Buffer bin format family Buffer Array array format family Array Map map format family Map (if usemap=true) Object (plain object) map format family Object (or Map if usemap=true) Object (see below) ext format family Object (see below) Note that both null and undefined are mapped to nil 0xC1 type. This means undefined value will be _upgraded_ to null in other words. Extension Types The MessagePack specification allows 128 application-specific extension types. The library uses the following types to make round-trip conversion possible for JavaScript native objects. TYPE OBJECT TYPE OBJECT 0x00 0x10 0x01 EvalError 0x11 Int8Array 0x02 RangeError 0x12 Uint8Array 0x03 ReferenceError 0x13 Int16Array 0x04 SyntaxError 0x14 Uint16Array 0x05 TypeError 0x15 Int32Array 0x06 URIError 0x16 Uint32Array 0x07 0x17 Float32Array 0x08 0x18 Float64Array 0x09 0x19 Uint8ClampedArray 0x0A RegExp 0x1A ArrayBuffer 0x0B Boolean 0x1B Buffer 0x0C String 0x1C 0x0D Date 0x1D DataView 0x0E Error 0x1E 0x0F Number 0x1F Other extension types are mapped to built-in ExtBuffer object. Custom Extension Types (Codecs) Register a custom extension type number to serialize/deserialize your own class instances. var msgpack = require("msgpack-lite"); var codec = msgpack.createCodec(); codec.addExtPacker(0x3F, MyVector, myVectorPacker); codec.addExtUnpacker(0x3F, myVectorUnpacker); var data = new MyVector(1, 2); var encoded = msgpack.encode(data, {codec: codec}); var decoded = msgpack.decode(encoded, {codec: codec}); function MyVector(x, y) { this.x = x; this.y = y; } function myVectorPacker(vector) { var array = ; return msgpack.encode(array); // return Buffer serialized } function myVectorUnpacker(buffer) { var array = msgpack.decode(buffer); return new MyVector(array, array); // return Object deserialized } The first argument of addExtPacker and addExtUnpacker should be an integer within the range of 0 and 127 (0x0 and 0x7F). myClassPacker is a function that accepts an instance of MyClass, and should return a buffer representing that instance. myClassUnpacker is the opposite: it accepts a buffer and should return an instance of MyClass. If you pass an array of functions to addExtPacker or addExtUnpacker, the value to be encoded/decoded will pass through each one in order. This allows you to do things like this: codec.addExtPacker(0x00, Date, ); You can also pass the codec option to msgpack.Decoder(options), msgpack.Encoder(options), msgpack.createEncodeStream(options), and msgpack.createDecodeStream(options). If you wish to modify the default built-in codec, you can access it at msgpack.codec.preset. Custom Codec Options msgpack.createCodec() function accepts some options. It does NOT have the preset extension types defined when no options given. var codec = msgpack.createCodec(); preset: It has the preset extension types described above. var codec = msgpack.createCodec({preset: true}); safe: It runs a validation of the value before writing it into buffer. This is the default behavior for some old browsers which do not support ArrayBuffer object. var codec = msgpack.createCodec({safe: true}); useraw: It uses raw formats instead of bin and str. var codec = msgpack.createCodec({useraw: true}); int64: It decodes msgpack's int64/uint64 formats with int64-buffer object. var codec = msgpack.createCodec({int64: true}); binarraybuffer: It ties msgpack's bin format with ArrayBuffer object, instead of Buffer object. var codec = msgpack.createCodec({binarraybuffer: true, preset: true}); uint8array: It returns Uint8Array object when encoding, instead of Buffer object. var codec = msgpack.createCodec({uint8array: true}); usemap: Uses the global JavaScript Map type, if available, to unpack MessagePack map elements. var codec = msgpack.createCodec({usemap: true}); Compatibility Mode The compatibility mode respects for msgpack's old spec . Set true to useraw. // default mode handles both str and bin formats individually msgpack.encode("Aa"); // => (str format) msgpack.encode(new Buffer()); // => (bin format) msgpack.decode(new Buffer()); // => 'Aa' (String) msgpack.decode(new Buffer()); // => (Buffer) // compatibility mode handles only raw format both for String and Buffer var options = {codec: msgpack.createCodec({useraw: true})}; msgpack.encode("Aa", options); // => (raw format) msgpack.encode(new Buffer(), options); // => (raw format) msgpack.decode(new Buffer(), options); // => (Buffer) msgpack.decode(new Buffer(), options).toString(); // => 'Aa' (String) Repository * https://github.com/kawanet/msgpack-lite See Also * http://msgpack.org/ License The MIT License (MIT) Copyright (c) 2015-2016 Yusuke Kawasaki Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. LUDOCODE/MSGPACK-TOOLS HTTPS://GITHUB.COM/LUDOCODE/MSGPACK-TOOLS INTRODUCTION MSGPACK-TOOLS contains simple command-line utilities for converting from MessagePack to JSON and vice-versa. They support options for lax parsing, lossy conversions, pretty-printing, and base64 encoding. * msgpack2json -- Convert MessagePack to JSON * json2msgpack -- Convert JSON to MessagePack They can be used for dumping MessagePack from a file or web API to a human-readable format, or for converting hand-written or generated JSON to MessagePack. The lax parsing mode supports comments and trailing commas in JSON, making it possible to hand-write your app or game data in JSON and convert it at build-time to MessagePack. BUILD STATUS BRANCH TRAVIS-CI _master_ _develop_ EXAMPLES To view a MessagePack file in a human-readable format for debugging purposes: msgpack2json -di file.mp To convert a hand-written JSON file to a MessagePack file, ignoring comments and trailing commas, and allowing embedded base64 with a base64: prefix: json2msgpack -bli file.json -o file.mp To fetch MessagePack from a web API and view it in a human-readable format: curl 'http://example/api/url' | msgpack2json -d To view the MessagePack-equivalent encoding of a JSON string: $ echo '{"compact": true, "schema": 0}' | json2msgpack | hexdump -C 00000000 82 a7 63 6f 6d 70 61 63 74 c3 a6 73 63 68 65 6d |..compact..schem| 00000010 61 00 |a.| 00000012 To test a MessagePack-RPC server via netcat: $ echo '' | json2msgpack | nc -q1 localhost 18800 | msgpack2json -d INSTALLATION * ARCH LINUX: msgpack-tools in the AUR , e.g. yaourt -S msgpack-tools * MAC OS X (Homebrew ): brew install https://ludocode.github.io/msgpack-tools.rb * DEBIAN (UBUNTU, ETC.): .deb package for x86_64 in the latest release ; install with dpkg For other platforms, msgpack-tools must be built from source. Download the msgpack-tools tarball from the latest release page (not the "source code" archive generated by GitHub, but the actual release package.) msgpack-tools uses CMake. A configure wrapper is provided that calls CMake, so you can simply run the usual: ./configure && make && sudo make install If you are building from the repository, you will need md2man to generate the man pages. DIFFERENCES BETWEEN MESSAGEPACK AND JSON MessagePack is intended to be very close to JSON in supported features, so they can usually be transparently converted from one to the other. There are some differences, however, which can complicate conversions. These are the differences in what objects are representable in each format: * JSON keys must be strings. MessagePack keys can be any type, including maps and arrays. * JSON supports "bignums", i.e. integers of any size. MessagePack integers must fit within a 64-bit signed or unsigned integer. * JSON real numbers are specified in decimal scientific notation and can have arbitrary precision. MessagePack real numbers are in IEEE 754 standard 32-bit or 64-bit binary. * MessagePack supports binary and extension type objects. JSON does not support binary data. Binary data is often encoded into a base64 string to be embedded into a JSON document. * A JSON document can be encoded in UTF-8, UTF-16 or UTF-32, and the entire document must be in the same encoding. MessagePack strings are required to be UTF-8, although this is not enforced by many encoding/decoding libraries. By default, msgpack2json and json2msgpack convert in strict mode. If an object in the source format is not representable in the destination format, the converter aborts with an error. A lax mode is available which performs a "lossy" conversion, and base64 conversion modes are available to support binary data in JSON. RYBAKIT/MSGPACK.PHP HTTPS://GITHUB.COM/RYBAKIT/MSGPACK.PHP MSGPACK.PHP A pure PHP implementation of the MessagePack serialization format. FEATURES * Fully compliant with the latest MessagePack specification , including BIN, STR and EXT types * Supports streaming unpacking * Supports unsigned 64-bit integers handling * Supports object serialization * Fully tested * Relatively fast TABLE OF CONTENTS * Installation * Usage * Packing * Packing options * Unpacking * Unpacking options * Extensions * Type transformers * Exceptions * Tests * Performance * License INSTALLATION The recommended way to install the library is through Composer : composer require rybakit/msgpack USAGE Packing To pack values you can either use an instance of a Packer: use MessagePack\Packer; $packer = new Packer(); ... $packed = $packer->pack($value); or call a static method on the MessagePack class: use MessagePack\MessagePack; ... $packed = MessagePack::pack($value); In the examples above, the method pack automatically packs a value depending on its type. But not all PHP types can be uniquely translated to MessagePack types. For example, the MessagePack format defines map and array types, which are represented by a single array type in PHP. By default, the packer will pack a PHP array as a MessagePack array if it has sequential numeric keys, starting from 0 and as a MessagePack map otherwise: $mpArr1 = $packer->pack(); // MP array $mpArr2 = $packer->pack(); // MP array $mpMap1 = $packer->pack(); // MP map {0: 1, 2: 3} $mpMap2 = $packer->pack(); // MP map {1: 2, 2: 3} $mpMap3 = $packer->pack(); // MP map {a: 1, b: 2} However, sometimes you need to pack a sequential array as a MessagePack map. To do this, use the packMap method: $mpMap = $packer->packMap(); // {0: 1, 1: 2} Here is a list of type-specific packing methods: $packer->packNil(); // MP nil $packer->packBool(true); // MP bool $packer->packInt(42); // MP int $packer->packFloat(M_PI); // MP float $packer->packStr('foo'); // MP str $packer->packBin("\x80"); // MP bin $packer->packArray(); // MP array $packer->packMap(); // MP map $packer->packExt(1, "\xaa"); // MP ext > _Check the "Type transformers" section below on how to pack custom > types._ Packing options The Packer object supports a number of bitmask-based options for fine-tuning the packing process (defaults are in bold): NAME DESCRIPTION FORCE_STR Forces PHP strings to be packed as MessagePack UTF-8 strings FORCE_BIN Forces PHP strings to be packed as MessagePack binary data DETECT_STR_BIN Detects MessagePack str/bin type automatically FORCE_ARR Forces PHP arrays to be packed as MessagePack arrays FORCE_MAP Forces PHP arrays to be packed as MessagePack maps DETECT_ARR_MAP Detects MessagePack array/map type automatically FORCE_FLOAT32 Forces PHP floats to be packed as 32-bits MessagePack floats FORCE_FLOAT64 Forces PHP floats to be packed as 64-bits MessagePack floats > _The type detection mode (DETECT_STR_BIN/DETECT_ARR_MAP) adds some > overhead which can be noticed when you pack large (16- and 32-bit) > arrays or strings. However, if you know the value type in advance > (for example, you only work with UTF-8 strings or/and associative > arrays), you can eliminate this overhead by forcing the packer to > use the appropriate type, which will save it from running the > auto-detection routine. Another option is to explicitly specify the > value type. The library provides 2 auxiliary classes for this, Map > and Binary. Check the "Type transformers" section below for > details._ Examples: use MessagePack\Packer; use MessagePack\PackOptions; // pack PHP strings to MP strings, PHP arrays to MP maps // and PHP 64-bit floats (doubles) to MP 32-bit floats $packer = new Packer(PackOptions::FORCE_STR | PackOptions::FORCE_MAP | PackOptions::FORCE_FLOAT32); // pack PHP strings to MP binaries and PHP arrays to MP arrays $packer = new Packer(PackOptions::FORCE_BIN | PackOptions::FORCE_ARR); // these will throw MessagePack\Exception\InvalidOptionException $packer = new Packer(PackOptions::FORCE_STR | PackOptions::FORCE_BIN); $packer = new Packer(PackOptions::FORCE_FLOAT32 | PackOptions::FORCE_FLOAT64); Unpacking To unpack data you can either use an instance of a BufferUnpacker: use MessagePack\BufferUnpacker; $unpacker = new BufferUnpacker(); ... $unpacker->reset($packed); $value = $unpacker->unpack(); or call a static method on the MessagePack class: use MessagePack\MessagePack; ... $value = MessagePack::unpack($packed); If the packed data is received in chunks (e.g. when reading from a stream), use the tryUnpack method, which attempts to unpack data and returns an array of unpacked messages (if any) instead of throwing an InsufficientDataException: while ($chunk = ...) { $unpacker->append($chunk); if ($messages = $unpacker->tryUnpack()) { return $messages; } } If you want to unpack from a specific position in a buffer, use seek(): $unpacker->seek(42); // set position equal to 42 bytes $unpacker->seek(-8); // set position to 8 bytes before the end of the buffer To skip bytes from the current position, use skip(): $unpacker->skip(10); // set position to 10 bytes ahead of the current position Besides the above methods BufferUnpacker provides type-specific unpacking methods, namely: $unpacker->unpackNil(); // PHP null $unpacker->unpackBool(); // PHP bool $unpacker->unpackInt(); // PHP int $unpacker->unpackFloat(); // PHP float $unpacker->unpackStr(); // PHP UTF-8 string $unpacker->unpackBin(); // PHP binary string $unpacker->unpackArray(); // PHP sequential array $unpacker->unpackMap(); // PHP associative array $unpacker->unpackExt(); // PHP Ext class Unpacking options The BufferUnpacker object supports a number of bitmask-based options for fine-tuning the unpacking process (defaults are in bold): NAME DESCRIPTION BIGINT_AS_EXCEPTION Throws an exception on integer overflow BIGINT_AS_GMP Converts overflowed integers to GMP objects BIGINT_AS_STR Converts overflowed integers to strings > _1. The binary MessagePack format has unsigned 64-bit as its largest > integer data type, but PHP does not support such integers, which > means that an overflow can occur during unpacking._ > > _2. Make sure that the GMP > extension is enabled._ Examples: use MessagePack\BufferUnpacker; use MessagePack\UnpackOptions; $packedUint64 = "\xcf"."\xff\xff\xff\xff"."\xff\xff\xff\xff"; $unpacker = new BufferUnpacker($packedUint64); var_dump($unpacker->unpack()); // string(20) "18446744073709551615" $unpacker = new BufferUnpacker($packedUint64, UnpackOptions::BIGINT_AS_GMP); var_dump($unpacker->unpack()); // object(GMP) {...} $unpacker = new BufferUnpacker($packedUint64, UnpackOptions::BIGINT_AS_EXCEPTION); $unpacker->unpack(); // throws MessagePack\Exception\IntegerOverflowException Extensions To define application-specific types use the Ext class: use MessagePack\Ext; use MessagePack\MessagePack; $packed = MessagePack::pack(new Ext(42, "\xaa")); $ext = MessagePack::unpack($packed); var_dump($ext->type === 42); // bool(true) var_dump($ext->data === "\xaa"); // bool(true) Type transformers In addition to the basic types , the library provides functionality to serialize and deserialize arbitrary types. In order to support a custom type you need to create and register a transformer. The transformer should implement either or both the Packable and/or the Unpackable interface. The purpose of Packable transformers is to serialize a specific value to one of the basic MessagePack types. A good example of such a transformer is a MapTransformer that comes with the library. It serializes Map objects (which are simple wrappers around PHP arrays) to MessagePack maps. This is useful when you want to explicitly mark that a given PHP array must be packed as a MessagePack map, without triggering the type's auto-detection routine. > _More types and type transformers can be found in src/Type > and src/TypeTransformer > directories._ The implementation is trivial: namespace MessagePack\TypeTransformer; use MessagePack\Packer; use MessagePack\Type\Map; class MapTransformer implements Packable { public function pack(Packer $packer, $value) : ?string { return $value instanceof Map ? $packer->packMap($value->map) : null; } } Once MapTransformer is registered, you can pack Map objects: use MessagePack\Packer; use MessagePack\PackOptions; use MessagePack\Type\Map; use MessagePack\TypeTransformer\MapTransformer; $packer = new Packer(PackOptions::FORCE_ARR); $packer->registerTransformer(new MapTransformer()); $packed = $packer->pack(); Transformers implementing the Unpackable interface are intended for unpacking extension types . For example, the code below shows how to create a transformer that allows you to work transparently with DateTime objects: use MessagePack\BufferUnpacker; use MessagePack\Packer; use MessagePack\TypeTransformer\Packable; use MessagePack\TypeTransformer\Unpackable; class DateTimeTransformer implements Packable, Unpackable { private $type; public function __construct(int $type) { $this->type = $type; } public function getType() : int { return $this->type; } public function pack(Packer $packer, $value) : ?string { if (!$value instanceof \DateTimeInterface) { return null; } return $packer->packExt($this->type, $packer->packStr($value->format('Y-m-d\TH:i:s.uP')) ); } public function unpack(BufferUnpacker $unpacker, int $extLength) { return new \DateTimeImmutable($unpacker->unpackStr()); } } Register DateTimeTransformer for both the packer and the unpacker with a unique extension type (an integer from 0 to 127) and you are ready to go: use App\MessagePack\DateTimeTransformer; use MessagePack\BufferUnpacker; use MessagePack\Packer; $transformer = new DateTimeTransformer(42); $packer = new Packer(); $packer->registerTransformer($transformer); $unpacker = new BufferUnpacker(); $unpacker->registerTransformer($transformer); $packed = $packer->pack(new DateTimeImmutable()); $date = $unpacker->reset($packed)->unpack(); > _More type transformer examples can be found in the examples > directory._ EXCEPTIONS If an error occurs during packing/unpacking, a PackingFailedException or UnpackingFailedException will be thrown, respectively. In addition, there are two more exceptions that can be thrown during unpacking: * InsufficientDataException * IntegerOverflowException An InvalidOptionException will be thrown in case an invalid option (or a combination of mutually exclusive options) is used. TESTS Run tests as follows: vendor/bin/phpunit Also, if you already have Docker installed, you can run the tests in a docker container. First, create a container: ./dockerfile.sh | docker build -t msgpack - The command above will create a container named msgpack with PHP 7.2 runtime. You may change the default runtime by defining the PHP_RUNTIME environment variable: PHP_RUNTIME='php:7.3-cli' ./dockerfile.sh | docker build -t msgpack - > _See a list of various runtimes here > ._ Then run the unit tests: docker run --rm --name msgpack -v $(pwd):/msgpack -w /msgpack msgpack Performance To check performance, run: php -n -dpcre.jit=1 -dzend_extension=opcache.so -dopcache.enable_cli=1 tests/bench.php This command will output something like: Filter: MessagePack\Tests\Perf\Filter\ListFilter Rounds: 3 Iterations: 100000 ============================================= Test/Target Packer BufferUnpacker --------------------------------------------- nil .................. 0.0041 ........ 0.0158 false ................ 0.0046 ........ 0.0166 true ................. 0.0042 ........ 0.0158 7-bit uint #1 ........ 0.0066 ........ 0.0129 7-bit uint #2 ........ 0.0066 ........ 0.0131 7-bit uint #3 ........ 0.0067 ........ 0.0134 5-bit sint #1 ........ 0.0068 ........ 0.0153 5-bit sint #2 ........ 0.0068 ........ 0.0153 5-bit sint #3 ........ 0.0067 ........ 0.0152 8-bit uint #1 ........ 0.0088 ........ 0.0223 8-bit uint #2 ........ 0.0088 ........ 0.0226 8-bit uint #3 ........ 0.0090 ........ 0.0224 16-bit uint #1 ....... 0.0120 ........ 0.0304 16-bit uint #2 ....... 0.0125 ........ 0.0306 16-bit uint #3 ....... 0.0122 ........ 0.0301 32-bit uint #1 ....... 0.0140 ........ 0.0358 32-bit uint #2 ....... 0.0138 ........ 0.0358 32-bit uint #3 ....... 0.0141 ........ 0.0365 64-bit uint #1 ....... 0.0159 ........ 0.0361 64-bit uint #2 ....... 0.0157 ........ 0.0363 64-bit uint #3 ....... 0.0157 ........ 0.0362 8-bit int #1 ......... 0.0089 ........ 0.0251 8-bit int #2 ......... 0.0090 ........ 0.0259 8-bit int #3 ......... 0.0094 ........ 0.0255 16-bit int #1 ........ 0.0121 ........ 0.0304 16-bit int #2 ........ 0.0121 ........ 0.0306 16-bit int #3 ........ 0.0121 ........ 0.0305 32-bit int #1 ........ 0.0141 ........ 0.0364 32-bit int #2 ........ 0.0146 ........ 0.0377 32-bit int #3 ........ 0.0145 ........ 0.0369 64-bit int #1 ........ 0.0142 ........ 0.0359 64-bit int #2 ........ 0.0143 ........ 0.0360 64-bit int #3 ........ 0.0147 ........ 0.0360 64-bit int #4 ........ 0.0142 ........ 0.0358 64-bit float #1 ...... 0.0148 ........ 0.0363 64-bit float #2 ...... 0.0152 ........ 0.0351 64-bit float #3 ...... 0.0147 ........ 0.0353 fix string #1 ........ 0.0176 ........ 0.0143 fix string #2 ........ 0.0197 ........ 0.0234 fix string #3 ........ 0.0197 ........ 0.0250 fix string #4 ........ 0.0238 ........ 0.0254 8-bit string #1 ...... 0.0245 ........ 0.0333 8-bit string #2 ...... 0.0297 ........ 0.0345 8-bit string #3 ...... 0.0372 ........ 0.0343 16-bit string #1 ..... 0.0392 ........ 0.0411 16-bit string #2 ..... 3.2959 ........ 0.3091 32-bit string ........ 3.2918 ........ 0.3149 wide char string #1 .. 0.0203 ........ 0.0248 wide char string #2 .. 0.0262 ........ 0.0323 8-bit binary #1 ...... 0.0206 ........ 0.0309 8-bit binary #2 ...... 0.0217 ........ 0.0332 8-bit binary #3 ...... 0.0219 ........ 0.0335 16-bit binary ........ 0.0255 ........ 0.0405 32-bit binary ........ 0.3719 ........ 0.3212 fix array #1 ......... 0.0070 ........ 0.0147 fix array #2 ......... 0.0288 ........ 0.0445 fix array #3 ......... 0.0513 ........ 0.0599 16-bit array #1 ...... 0.1540 ........ 0.1718 16-bit array #2 ........... S ............. S 32-bit array .............. S ............. S complex array ........ 0.2138 ........ 0.2679 fix map #1 ........... 0.0999 ........ 0.1239 fix map #2 ........... 0.0429 ........ 0.0507 fix map #3 ........... 0.0503 ........ 0.0721 fix map #4 ........... 0.0458 ........ 0.0604 16-bit map #1 ........ 0.2516 ........ 0.3197 16-bit map #2 ............. S ............. S 32-bit map ................ S ............. S complex map .......... 0.2972 ........ 0.3501 fixext 1 ............. 0.0146 ........ 0.0403 fixext 2 ............. 0.0148 ........ 0.0422 fixext 4 ............. 0.0148 ........ 0.0426 fixext 8 ............. 0.0165 ........ 0.0427 fixext 16 ............ 0.0156 ........ 0.0422 8-bit ext ............ 0.0197 ........ 0.0494 16-bit ext ........... 0.0234 ........ 0.0576 32-bit ext ........... 0.3674 ........ 0.3327 ============================================= Total 9.4513 4.5950 Skipped 4 4 Failed 0 0 Ignored 0 0 You may change default benchmark settings by defining the following environment variables: * MP_BENCH_TARGETS (pure_p, pure_ps, pure_pa, pure_psa, pure_bu, pecl_p, pecl_u) * MP_BENCH_ITERATIONS/MP_BENCH_DURATION * MP_BENCH_ROUNDS * MP_BENCH_TESTS For example: export MP_BENCH_TARGETS=pure_p export MP_BENCH_ITERATIONS=1000000 export MP_BENCH_ROUNDS=5 # a comma separated list of test names export MP_BENCH_TESTS='complex array, complex map' # or a group name # export MP_BENCH_TESTS='-@slow' // @pecl_comp # or a regexp # export MP_BENCH_TESTS='/complex (array|map)/' php -n -dpcre.jit=1 -dzend_extension=opcache.so -dopcache.enable_cli=1 tests/bench.php Another example, benchmarking both the library and the msgpack pecl extension : MP_BENCH_TARGETS=pure_ps,pure_bu,pecl_p,pecl_u \ php -n -dpcre.jit=1 -dextension=msgpack.so -dzend_extension=opcache.so -dopcache.enable_cli=1 tests/bench.php Output: Filter: MessagePack\Tests\Perf\Filter\ListFilter Rounds: 3 Iterations: 100000 ====================================================================================== Test/Target Packer (force_str) BufferUnpacker msgpack_pack msgpack_unpack -------------------------------------------------------------------------------------- nil ............................. 0.0040 ........ 0.0158 ...... 0.0065 ........ 0.0052 false ........................... 0.0046 ........ 0.0158 ...... 0.0064 ........ 0.0051 true ............................ 0.0045 ........ 0.0158 ...... 0.0065 ........ 0.0051 7-bit uint #1 ................... 0.0066 ........ 0.0130 ...... 0.0065 ........ 0.0054 7-bit uint #2 ................... 0.0067 ........ 0.0129 ...... 0.0065 ........ 0.0052 7-bit uint #3 ................... 0.0065 ........ 0.0129 ...... 0.0065 ........ 0.0052 5-bit sint #1 ................... 0.0067 ........ 0.0153 ...... 0.0065 ........ 0.0054 5-bit sint #2 ................... 0.0067 ........ 0.0152 ...... 0.0065 ........ 0.0053 5-bit sint #3 ................... 0.0068 ........ 0.0154 ...... 0.0065 ........ 0.0054 8-bit uint #1 ................... 0.0087 ........ 0.0223 ...... 0.0066 ........ 0.0057 8-bit uint #2 ................... 0.0088 ........ 0.0223 ...... 0.0066 ........ 0.0057 8-bit uint #3 ................... 0.0087 ........ 0.0222 ...... 0.0067 ........ 0.0057 16-bit uint #1 .................. 0.0120 ........ 0.0295 ...... 0.0068 ........ 0.0057 16-bit uint #2 .................. 0.0118 ........ 0.0296 ...... 0.0069 ........ 0.0057 16-bit uint #3 .................. 0.0119 ........ 0.0297 ...... 0.0068 ........ 0.0057 32-bit uint #1 .................. 0.0138 ........ 0.0358 ...... 0.0067 ........ 0.0056 32-bit uint #2 .................. 0.0137 ........ 0.0358 ...... 0.0066 ........ 0.0058 32-bit uint #3 .................. 0.0138 ........ 0.0358 ...... 0.0068 ........ 0.0057 64-bit uint #1 .................. 0.0140 ........ 0.0362 ...... 0.0067 ........ 0.0056 64-bit uint #2 .................. 0.0140 ........ 0.0364 ...... 0.0067 ........ 0.0056 64-bit uint #3 .................. 0.0141 ........ 0.0362 ...... 0.0066 ........ 0.0057 8-bit int #1 .................... 0.0089 ........ 0.0251 ...... 0.0068 ........ 0.0057 8-bit int #2 .................... 0.0090 ........ 0.0252 ...... 0.0066 ........ 0.0056 8-bit int #3 .................... 0.0089 ........ 0.0251 ...... 0.0066 ........ 0.0056 16-bit int #1 ................... 0.0120 ........ 0.0304 ...... 0.0067 ........ 0.0057 16-bit int #2 ................... 0.0120 ........ 0.0314 ...... 0.0069 ........ 0.0059 16-bit int #3 ................... 0.0124 ........ 0.0314 ...... 0.0070 ........ 0.0058 32-bit int #1 ................... 0.0144 ........ 0.0372 ...... 0.0067 ........ 0.0056 32-bit int #2 ................... 0.0139 ........ 0.0364 ...... 0.0068 ........ 0.0056 32-bit int #3 ................... 0.0139 ........ 0.0364 ...... 0.0067 ........ 0.0056 64-bit int #1 ................... 0.0143 ........ 0.0363 ...... 0.0067 ........ 0.0058 64-bit int #2 ................... 0.0142 ........ 0.0363 ...... 0.0068 ........ 0.0057 64-bit int #3 ................... 0.0143 ........ 0.0366 ...... 0.0067 ........ 0.0058 64-bit int #4 ................... 0.0142 ........ 0.0364 ...... 0.0067 ........ 0.0057 64-bit float #1 ................. 0.0148 ........ 0.0353 ...... 0.0066 ........ 0.0057 64-bit float #2 ................. 0.0152 ........ 0.0356 ...... 0.0066 ........ 0.0057 64-bit float #3 ................. 0.0147 ........ 0.0355 ...... 0.0066 ........ 0.0057 fix string #1 ................... 0.0082 ........ 0.0143 ...... 0.0069 ........ 0.0057 fix string #2 ................... 0.0101 ........ 0.0234 ...... 0.0069 ........ 0.0070 fix string #3 ................... 0.0101 ........ 0.0256 ...... 0.0071 ........ 0.0072 fix string #4 ................... 0.0102 ........ 0.0255 ...... 0.0068 ........ 0.0067 8-bit string #1 ................. 0.0122 ........ 0.0335 ...... 0.0068 ........ 0.0068 8-bit string #2 ................. 0.0127 ........ 0.0337 ...... 0.0069 ........ 0.0069 8-bit string #3 ................. 0.0127 ........ 0.0337 ...... 0.0102 ........ 0.0069 16-bit string #1 ................ 0.0161 ........ 0.0403 ...... 0.0103 ........ 0.0074 16-bit string #2 ................ 0.3583 ........ 0.3081 ...... 0.3529 ........ 0.2745 32-bit string ................... 0.3609 ........ 0.3168 ...... 0.3511 ........ 0.2749 wide char string #1 ............. 0.0098 ........ 0.0248 ...... 0.0068 ........ 0.0070 wide char string #2 ............. 0.0122 ........ 0.0324 ...... 0.0069 ........ 0.0069 8-bit binary #1 ...................... I ............. I ........... F ............. I 8-bit binary #2 ...................... I ............. I ........... F ............. I 8-bit binary #3 ...................... I ............. I ........... F ............. I 16-bit binary ........................ I ............. I ........... F ............. I 32-bit binary ........................ I ............. I ........... F ............. I fix array #1 .................... 0.0070 ........ 0.0146 ...... 0.0152 ........ 0.0069 fix array #2 .................... 0.0292 ........ 0.0444 ...... 0.0175 ........ 0.0165 fix array #3 .................... 0.0422 ........ 0.0617 ...... 0.0196 ........ 0.0209 16-bit array #1 ................. 0.1536 ........ 0.1716 ...... 0.0321 ........ 0.0449 16-bit array #2 ...................... S ............. S ........... S ............. S 32-bit array ......................... S ............. S ........... S ............. S complex array ........................ I ............. I ........... F ............. F fix map #1 ........................... I ............. I ........... F ............. I fix map #2 ...................... 0.0331 ........ 0.0502 ...... 0.0171 ........ 0.0188 fix map #3 ........................... I ............. I ........... F ............. I fix map #4 ........................... I ............. I ........... F ............. I 16-bit map #1 ................... 0.2506 ........ 0.3160 ...... 0.0330 ........ 0.0689 16-bit map #2 ........................ S ............. S ........... S ............. S 32-bit map ........................... S ............. S ........... S ............. S complex map ..................... 0.2679 ........ 0.3518 ...... 0.0659 ........ 0.0753 fixext 1 ............................. I ............. I ........... F ............. F fixext 2 ............................. I ............. I ........... F ............. F fixext 4 ............................. I ............. I ........... F ............. F fixext 8 ............................. I ............. I ........... F ............. F fixext 16 ............................ I ............. I ........... F ............. F 8-bit ext ............................ I ............. I ........... F ............. F 16-bit ext ........................... I ............. I ........... F ............. F 32-bit ext ........................... I ............. I ........... F ............. F ====================================================================================== Total 2.0227 2.9568 1.2268 1.0765 Skipped 4 4 4 4 Failed 0 0 17 9 Ignored 17 17 0 8 > _Note that the msgpack extension (0.5.2+, 2.0) doesn't support EXT, > BIN and UTF-8 STR types._ LICENSE The library is released under the MIT License. See the bundled LICENSE file for details. MSGPACK4Z/MSGPACK4Z-CIRCE HTTPS://GITHUB.COM/MSGPACK4Z/MSGPACK4Z-CIRCE MSGPACK4Z-CIRCE * https://msgpack.org/ * https://github.com/msgpack/msgpack/blob/master/spec.md latest stable version libraryDependencies += "com.github.xuwei-k" %% "msgpack4z-circe" % "0.8.0" * sxr snapshot version libraryDependencies += "com.github.xuwei-k" %% "msgpack4z-circe" % "0.8.1-SNAPSHOT" resolvers += Opts.resolver.sonatypeSnapshots * sxr MICHAEL-YUJI/YSMESSAGEPACK HTTPS://GITHUB.COM/MICHAEL-YUJI/YSMESSAGEPACK YSMESSAGEPACK- FOR SWIFT 3 YSMessagePack is a messagePack packer/unpacker written in swift (swift 3 ready). It is designed to be easy to use. YSMessagePack include following features: * Pack custom structs and classes / unpack objects by groups and apply handler to each group (easier to re-construct your struct$) * Asynchronous unpacking * Pack and unpack multiple message-packed data regardless of types with only one line of code * Specify how many items to unpack * Get remaining bytes that were not message-packed ; start packing from some index -- so you can mix messagepack with other protocol!!! * Helper methods to cast NSData to desired types * Operator +^ and +^= to join NSData VERSION 1.6.2 (Dropped swift 2 support, swift 3 support only from now on) INSTALLATION * Simply add files under YSMessagePack/Classes to your project, * use cocoapod, add "pod 'YSMessagePack', '~> 1.6.2' to your podfile USAGE Pack: let exampleInt: Int = 1 let exampleStr: String = "Hello World" let exampleArray: = let bool: Bool = true // To pack items, just put all of them in a single array // and call the `pack(items:)` function //this will be the packed data let msgPackedBytes: NSData = pack(items: ) // Now your payload is ready to send!!! But what if we have some custom data structure to send? //To make your struct / class packable struct MyStruct: Packable { //Confirm to this protocol var name: String var index: Int func packFormat() -> { //protocol function return //pack order } func msgtype() -> MsgPackTypes { return .Custom } } let exampleInt: Int = 1 let exampleStr: String = "Hello World" let exampleArray: = let bool: Bool = true let foo = MyStruct(name: "foo", index: 626) let msgPackedBytes = pack(items: ) OR YOU CAN PACK THEM INDIVIDUALLY AND ADD THEM TO A BYTE ARRAY MANUALLY (WHICH IS ALSO LESS EXPENSIVE) let exampleInt: Int = 1 let exampleStr: String = "Hello World" let exampleArray: = //Now pack them individually let packedInt = exampleInt.packed() //if you didn't specific encoding, the default encoding will be ASCII #if swift(>=3) let packedStr = exampleStr.packed(withEncoding: NSASCIIStringEncoding) #else let packedStr = exampleStr.packed(withEncoding: .ascii) #endif let packedArray = exampleArray.packed() //You can use this operator +^ the join the data on rhs to the end of data on lhs let msgPackedBytes: NSData = packedInt +^ packedStr +^ packedArray UNPACK YSMessagePack offer a number of different ways and options to unpack include unpack asynchronously, see the example project for detail. To unpack a messagepacked bytearray is pretty easy: do { //The unpack method will return an array of NSData which each element is an unpacked object let unpackedItems = try msgPackedBytes.itemsUnpacked() //instead of casting the NSData to the type you want, you can call these `.castTo..` methods to do the job for you let int: Int = unpackedItems.castToInt() //Same as packing, you can also specify the encoding you want to use, default is ASCII let str: String = unpackedItem.castToString() let array: NSArray = unpackedItems.castToArray() } catch let error as NSError{ NSLog("Error occurs during unpacking: %@", error) } //Remember how to pack your struct? Here is a better way to unpack a stream of bytes formatted in specific format let testObj1 = MyStruct(name: "TestObject1", index: 1) let testObj2 = MyStruct(name: "TestObject2", index: 2) let testObj3 = MyStruct(name: "TestObject3", index: 3) let packed = packCustomObjects(testObj1, testObj2, testObj3) //This is an other method that can pack your own struct easier let nobjsInOneGroup = 2 try! packed.unpackByGroupsWith(nobjsInOneGroup) { (unpackedData, isLast) -> Bool //you can also involve additional args like number of groups to unpack guard let name = unpackedData.castToString() else {return false} //abort unpacking hen something wrong let index = unpackedData let testObj = MyStruct(name: name, index: index) // assembly return true //proceed unpacking, or return false to abort } IF YOU DON'T WANT TO UNPACK EVERY SINGLE THING INCLUDED IN THE MESSAGE-PACK BYTE ARRAY, YOU CAN ALSO SPECIFY AN AMOUNT TO UNPACK, IF YOU WANT TO KEEP THE REMAINING BYTES, YOU CAN PUT TRUE IN THE RETURNREMAININGBYTES ARGUMENT, THE REMAINING BYTES WILL STORED IN THE END OF THE NSDATA ARRAY. do { //Unpack only 2 objects, and we are not interested in remaining bytes let unpackedItems = try msgPackedBytes.itemsUnpacked(specific_amount: 2, returnRemainingBytes: false) print(unpackedItems.count) //will print 2 } catch let error as NSError{ NSLog("Error occurs during unpacking: %@", error) } CAESAY/MPACK HTTPS://GITHUB.COM/CAESAY/MPACK MPACK This library is a lightweight implementation of the MessagePack binary serialization format. MessagePack is a 1-to-1 binary representation of JSON, and the official specification can be found here: https://github.com/msgpack... . (https://ci.appveyor.com/project/caesay/mpack) NOTES * This library is designed to be super light weight. * Its easiest to understand how this library works if you think in terms of json. The type MPackMap represents a dictionary, and the type MPackArray represents an array. * Create MPack instances with the static method MPack.From(object);. You can pass any simple type (such as string, integer, etc), or any Array composed of a simple type. MPack also has implicit conversions from most of the basic types built in. * Transform an MPack object back into a CLR type with the static method MPack.To(); or MPack.To(type);. MPack also has EXPLICIT converions going back to most basic types, you can do string str = (string)mpack; for instance. * MPack now supports native asynchrounous reading and cancellation tokens. It will _not_ block a thread to wait on a stream. NUGET MPack is available as a NuGet package! PM> Install-Package MPack USAGE Create a object model that can be represented as MsgPack. Here we are creating a dictionary, but really it can be anything: MPackMap dictionary = new MPackMap { { "array1", MPack.From(new { "array1_value1", // implicitly converted string MPack.From("array1_value2"), MPack.FromString("array1_value3"), }) }, {"bool1", MPack.From(true)}, //boolean {"double1", MPack.From(50.5)}, //single-precision float {"double2", MPack.From(15.2)}, {"int1", 50505}, // implicitly converted integer {"int2", MPack.From(50)} // integer }; Serialize the data to a byte array or to a stream to be saved, transmitted, etc: byte encodedBytes = dictionary.EncodeToBytes(); // -- or -- dictionary.EncodeToStream(stream); Parse the binary data back into a MPack object model (you can also cast back to an MPackMap or MPackArray after reading if you want dictionary/array methods): var reconstructed = MPack.ParseFromBytes(encodedBytes); // -- or -- var reconstructed = MPack.ParseFromStream(stream); Turn MPack objects back into types that we understand with the generic To<>() method. Since we know the types of everything here we can just call To() to reconstruct our bool, but if you don't know you can access the instance enum MPack.ValueType to know what kind of value it is: bool bool1 = reconstructed.To(); var array1 = reconstructed as MPackArray; var array1_value1 = array1; double double1 = reconstructed.To(); //etc... CREDITS The following people/projects have made this possible: * Me: atdot * All of the people that make MessagePack happen: https://github.com/msgpack JONATHONL/GOODFORM HTTPS://GITHUB.COM/JONATHONL/GOODFORM GOODFORM Form validation library. Includes MsgPack and JSON serializer/deserializer. BUILDING GoodForm uses std::any, which requires c++17. When c++17 is not available, boost::any is expected and will be installed automatically when using cget . cd goodform cget install -f ./requirements.txt # Install dependencies locally. mkdir build && cd build # Create out of source build directory. cmake -DCMAKE_TOOLCHAIN_FILE=../cget/cget/cget.cmake .. # Configure project with dependency paths. make MSGPACK USAGE std::stringstream ss; goodform::any var, var2; var = goodform::object { {"compact", true}, {"schema", 0} }; goodform::msgpack::serialize(var, ss); goodform::msgpack::deserialize(ss, var2); goodform::form form(var2); struct { bool compact; std::int32_t schema; } mpack; mpack.compact = form.at("compact").boolean().val(); mpack.schema = form.at("schema").int32().val(); if (form.is_good()) { std::cout << "{ \"compact\": " << std::boolalpha << mpack.compact << ", \"schema\": " << mpack.schema << " }" << std::endl; } JSON USAGE goodform::any var; std::stringstream ss; ss << "{" << std::endl << "\"first_name\":\"John\", // This is a comment" << std::endl << "\"last_name\":\"Smith\", " << std::endl << "\"age\": 23," << std::endl << "\"gpa\": 4.0," << std::endl << "\"email\":\"john.smith@example.com\"," << std::endl << "\"password_hash\":\"5f4dcc3b5aa765d61d8327deb882cf99\"," << std::endl << "\"interests\": " << std::endl << "}" << std::endl; goodform::json::deserialize(ss, var); goodform::form form(var); struct { std::string first_name; std::string last_name; std::uint8_t age; float gpa; std::string email; std::string password_hash; bool subscribe_to_email_marketing; std::list interests; } form_data; form_data.first_name = form.at("first_name").string().match(std::regex("^{1,64}$")).val(); form_data.last_name = form.at("last_name").string().match(std::regex("^{1,64}$")).val(); form_data.age = form.at("age").uint8().val(); form_data.gpa = form.at("gpa").float32().gte(0).lte(4.0).val(); form_data.email = form.at("email").string().match(std::regex("^.{3,256}$")).val(); form_data.password_hash = form.at("password_hash").string().match(std::regex("^{32}$")).val(); form_data.subscribe_to_email_marketing = form.at("subscribe_to_email_marketing", true).boolean().val(); // Optional field defaults to true. form.at("interests").array().for_each((goodform::sub_form& sf, std::size_t i) { form_data.interests.push_back(sf.string().val()); }); if (form.is_good()) { // Use validated form_data. } else { // Handle error. } HEADS-PROJECT/ARDUINO_MSGPACK HTTPS://GITHUB.COM/HEADS-PROJECT/ARDUINO_MSGPACK ARDUINO_MSGPACK This Arduino library provides a light weight serializer and parser for messagepack. INSTALL Download the zip, and import it with your Arduino IDE: _Sketch>Include Library>Add .zip library_ USAGE See the either the .h file, or the examples (led_controller and test_uno_writer). In short: * functions like msgpck_what_next(Stream * s); watch the next type of data without reading it (without advancing the buffer of Stream s). * functions like msgpck_read_bool(Stream * s, bool *b) read a value from Stream s. * functions like msgpck_write_bool(Stream * s, bool b) write a value on Stream s. Notes: * Stream are used as much as possible in order not to add to much overhead with buffers. Therefore you should be able to store the minimum number of value at a given time. * Map and Array related functions concern only their headers. Ex: If you want to write an array containing two elements you should write the array header, then write the two elements. LIMITATIONS Currently the library does NOT support: * 8 bytes float (Only 4 bytes floats are supported by default on every Arduino and floats are anyway not recommended on Arduino) * 2^32 char long (or longer) strings * 2^32 byte long (or longer) bins * extention types. POLYGLOTTED/MSGPACK-PYTHON HTTPS://GITHUB.COM/POLYGLOTTED/MSGPACK-PYTHON msgpack ST3V0/MSGPACK-AS3 HTTPS://GITHUB.COM/ST3V0/MSGPACK-AS3 MESSAGEPACK FOR ACTIONSCRIPT3 msgpack-as3 is an implementation of the latest MessagePack specification for ActionScript3 language (Flash, Flex and AIR). * Message Pack specification (Dec 22, 2015): https://github.com/msgpack/msgpack/blob/master/spec.md BASIC USAGE Serialize and Deserialize The usage of MsgPack class is very simple. You need create an object and call read and write methods. ```actionscript // message pack object created var msgpack:MsgPack = new MsgPack(); // encode an array var bytes:ByteArray = msgpack.write(); // rewind the buffer bytes.position = 0; // print the decoded object trace(msgpack.read(bytes)); ### Flags Currently there are three flags which you may use to initialize a MsgPack object:
* MsgPackFlags.READ_STRING_AS_BYTE_ARRAY: message pack string data is read as byte array instead of string; * MsgPackFlags.ACCEPT_LITTLE_ENDIAN: MsgPack objects will work with little endian buffers (message pack specification defines big endian as default). * MsgPackFlags.SPEC2013_COMPATIBILITY: MsgPack will run in backwards compatibility mode. ```actionscript var msg:MsgPack; // use logical operator OR to set the flags. msgpack = new MsgPack(MsgPackFlags.READ_STRING_AS_BYTE_ARRAY | MsgPackFlags.ACCEPT_LITTLE_ENDIAN); ADVANCED USAGE Extensions You can create your own Extension Workers by extending the ExtensionWorker Class and then assigning it to the MsgPack Factory. The following example assigns a custom worker which extends the ExtensionWorker Class. ```actionscript var msgpack:MsgPack = new MsgPack(); // Assign the new worker to the factory. msgpack.factory.assign(new CustomWorker()); For more information regarding Extensions refer to the MessagePack specification.
### Priorities Worker priority behaves similar to how the Adobe Event Dispatcher priorities work. In MessagePack, deciding which worker will be use for serializing/deserializing depends on two(2) factors.
1. The order in which the worker was assigned to the factory. 2. The priority of the worker. Higher values take precedence. All workers have a default priority of 0. In the following example workerB will never be used because it's assign after workerA
```actionscript var msgpack:MsgPack = new MsgPack(); var workerA:StringWorker = new StringWorker(); var workerB:DifferentStringWorker = new DifferentStringWorker(); msgpack.factory.assign(workerA); msgpack.factory.assign(workerB); However if we adjust the priority of workerB, then workerA will never be used. ```actionscript var msgpack:MsgPack = new MsgPack(); var workerA:StringWorker = new StringWorker(); var workerB:DifferentStringWorker = new DifferentStringWorker(null, 1); msgpack.factory.assign(workerA); msgpack.factory.assign(workerB); ## Credits This application uses Open Source components. You can find the source code of their open source projects along with license information below. We acknowledge and are grateful to these developers for their contributions to open source. Project: as3-msgpack https://github.com/loteixeira/as3-msgpack Copyright (C) 2013 Lucas Teixeira License (Apache V2.0) http://www.apache.org/licenses/LICENSE-2.0 EDDELBUETTEL/RCPPMSGPACK HTTPS://GITHUB.COM/EDDELBUETTEL/RCPPMSGPACK RCPPMSGPACK Convert to and from msgpack objects in R using the official msgpack-c API through Rcpp. _A flowchart describing the conversion of R objects into msgpack objects and back._ Msgpack EXT types are converted to raw vectors with EXT attributes containing the extension type. The extension type must be an integer from 0 to 127. Maps are converted to data.frames with additional class "map". Map objects in R contain key and value list columns and can be simplified to named lists or named vectors. The helper function msgpack_map creates map objects that can be serialized into msgpack. For more information on msgpack types, see here . Installation: * install.packages("RcppMsgPack") Example: See tests/tests.r for more examples. library(RcppMsgPack) library(microbenchmark) x <- as.list(1:1e6) microbenchmark(xpk <- msgpack_pack(x), times=3) microbenchmark(xu <- msgpack_unpack(xpk), times=3) STEAKKNIFE/MSGPACK.CR HTTPS://GITHUB.COM/STEAKKNIFE/MSGPACK.CR MSGPACK.CR A low-level msgpack codec for Crystal TODO * More specs * Mapping INSTALLATION Add this to your application's shard.yml: dependencies: msgpack: github: steakknife/msgpack.cr USAGE require "msgpack" 1.to_msgpack # => Slice # write 2_i32 to file foo.msgpack File.open("foo.msgpack", "w") { |f| f.write(2.to_msgpack) } EXTENDING Encoding Any type can become encodable by including Msgpack::Encodable and defining to_msgpack(io : IO) Decoding Any type can become decodable by following the example DEVELOPMENT Run tests crystal spec ALTERNATE IMPLEMENTATIONS * github: benoist/msgpack-crystal CONTRIBUTING * Fork it ( https://github.com/steakknife/msgpack.cr/fork ) * Create your feature branch (git checkout -b my-new-feature) * Commit your changes (git commit -am 'Add some feature') * Push to the branch (git push origin my-new-feature) * Create a new Pull Request CONTRIBUTORS * Barry Allard - creator, maintainer LICENSE MIT COPYRIGHT 2016 (c) Copyright Barry Allard AR90N/MSGPACK11 HTTPS://GITHUB.COM/AR90N/MSGPACK11 WHAT IS MSGPACK11 ? msgpack11 is a tiny MsgPack library for C++11, providing MsgPack parsing and serialization. This library is inspired by json11 . The API of msgpack11 is designed to be similar with json11. INSTALLATION * Using CMake git clone git@github.com:ar90n/msgpack11.git mkdir build cd build cmake ../msgpack11 make && make install * Using Buck git clone git@github.com:ar90n/msgpack11.git cd msgpack11 buck build :msgpack11 EXAMPLE MsgPack my_msgpack = MsgPack::object { { "key1", "value1" }, { "key2", false }, { "key3", MsgPack::array { 1, 2, 3 } }, }; //access to elements std::cout << my_msgpack.string_value(); //serialize std::string msgpack_bytes = my_msgpack.dump(); //deserialize std::string err; MsgPack des_msgpack = MsgPack::parse(msgpack_bytes, err); There are more specific examples in example.cpp. Please see it. BENCHMARK Derived from schemaless-benchmarks LIBRARY BINARY SIZE TIME @ SMALLEST TIME @ SMALL TIME @ MEDIUM TIME @ LARGE TIME @ LARGEST msgpack-c-pack(v2.1.4) 6649 0.55 2.38 43.22 711.75 8748.20 msgpack-c-unpack(v2.1.4) 21804 1.34 6.00 83.09 714.64 11192.32 msgpack11-pack(v0.0.9) 99844 20.80 130.04 1063.24 10466.65 136640.99 msgpack11-unpack(v0.0.9) 99460 13.31 92.54 786.73 7345.43 99119.56 CPU : 2.6 GHz Intel Core i7 Memory : 16 GB 2133 MHz LPDDR3 Git revision : 6f6b4302b68b3c88312eb24367418b7fce81298c FEATURE * Support serialization and deserialization. ACKNOWLEDGEMENT * json11 * msgpack-c * schemaless-benchmarks LICENSE This software is released under the MIT License, see LICENSE.txt. PIERRE-VIGIER/PERL6-DATA-MESSAGEPACK HTTPS://GITHUB.COM/PIERRE-VIGIER/PERL6-DATA-MESSAGEPACK PERL6-DATA-MESSAGEPACK NAME Data::MessagePack - Perl 6 implementation of MessagePack SYNOPSIS use Data::MessagePack; my $data-structure = { key => 'value', k2 => }; my $packed = Data::MessagePack::pack( $data-structure ); my $unpacked = Data::MessagePack::unpack( $packed ); Or for streaming: use Data::MessagePack::StreamingUnpacker; my $supplier = Some Supplier; #Could be from IO::Socket::Async for instance my $unpacker = Data::MessagePack::StreamingUnpacker.new( source => $supplier.Supply ); $unpacker.tap( -> $value { say "Got new value"; say $value.perl; }, done => { say "Source supply is done"; } ); DESCRIPTION The present module proposes an implemetation of the MessagePack specification as described on http://msgpack.org/ . The implementation is now in Pure Perl which could come as a performance penalty opposed to some other packer implemented in C. WHY THAT MODULE There are already some part of MessagePack implemented in Perl6, with for instance MessagePack available here: https://github.com/uasi/messagepack-pm6, however that module only implements the unpacking part of the specification. Futhermore, that module uses the unpack functionality which is tagged as experimental as of today FUNCTIONS FUNCTION PACK That function takes a data structure as parameter, and returns a Blob with the packed version of the data structure. FUNCTION UNPACK That function takes a MessagePack packed message as parameter, and returns the deserialized data structure. AUTHOR Pierre VIGIER CONTRIBUTORS Timo Paulssen LICENSE Artistic License 2.0 TAGOMORIS/MSGPACK-INSPECT HTTPS://GITHUB.COM/TAGOMORIS/MSGPACK-INSPECT MSGPACK-INSPECT This is a command line tool to inspect/show a data serialized by MessagePack . INSTALLATION Executable binary files are available from releases . Download a file for your platform, and use it. Otherwise, you can install rubygem version on your CRuby runtime: $ gem install msgpack-inspect USAGE Usage: msgpack-inspect FILE Options: -f, --format FORMAT output format of inspection result (yaml/json/jsonl) -r, --require LIB ruby file path to require (to load ext type definitions) -v, --version Show version of this software -h, --help Show this message -r option is available oly with rubygem version, and unavailable with mruby binary release. FILE is a file which msgpack binary stored. Specify - to inspect data from STDIN. This command shows the all data contained in specified format (YAML in default). --- - format: "false" header: "0xc2" data: "0xc2" value: false - format: "true" header: "0xc3" data: "0xc3" value: true EXAMPLE This is an example to inspect a data from STDIN. The data corresponds to {"compact":true,"schema":0} in JSON. $ printf "\x82\xa7compact\xc3\xa6schema\x00" | msgpack-inspect - --- - format: "fixmap" header: "0x82" length: 2 children: - key: format: "fixstr" header: "0xa7" length: 7 data: "0x636f6d70616374" value: "compact" value: format: "true" header: "0xc3" data: "0xc3" value: true - key: format: "fixstr" header: "0xa6" length: 6 data: "0x736368656d61" value: "schema" value: format: "fixint" header: "0x00" data: "0x00" value: 0 TODO: show more example CONTRIBUTING Bug reports and pull requests are welcome on GitHub at . TRIS-FOUNDATION/MESSAGEPACK HTTPS://GITHUB.COM/TRIS-FOUNDATION/MESSAGEPACK MESSAGEPACK MESSAGEPACK is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON. But it's faster and smaller. Small integers are encoded into a single byte, and typical short strings require only one extra byte in addition to the strings themselves. PACKAGE.SWIFT .package(url: "https://github.com/tris-foundation/messagepack.git", .branch("master")) MEMO public enum MessagePack { case `nil` case int(Int) case uint(UInt) case bool(Bool) case float(Float) case double(Double) case string(String) case binary() case array() case map() case extended(Extended) public struct Extended { public let type: Int8 public let data: public init(type: Int8, data: ) { self.type = type self.data = data } } } USAGE You can find this code and more in examples . Basic API let hey = MessagePack("hey there!") let bytes = try MessagePack.encode(hey) let original = String(try MessagePack.decode(bytes: bytes)) Stream API let hey = MessagePack("hey there!") let stream = BufferedStream(stream: NetworkStream(socket: client)) try MessagePack.encode(hey, to: stream) try stream.flush() let original = String(try MessagePack.decode(from: stream)) Performance optimized let output = OutputByteStream() var encoder = MessagePackWriter(output) try encoder.encode("one") try encoder.encode(2) try encoder.encode(3.0) let encoded = output.bytes var decoder = MessagePackReader(InputByteStream(encoded)) let string = try decoder.decode(String.self) let int = try decoder.decode(UInt8.self) let double = try decoder.decode(Double.self) print("decoded manually: \(string), \(int), \(double)") CLWI/CWPACK HTTPS://GITHUB.COM/CLWI/CWPACK CWPACK CWPack is a lightweight and yet complete implementation of the MessagePack serialization format version 5 . EXCELLENT PERFORMANCE Together with MPack , CWPack is the fastest open-source messagepack implementation. Both totally outperform CMP and msgpack-c DESIGN CWPack does no memory allocations and no file handling. All that is done outside of CWPack. CWPack is working against memory buffers. User defined handlers are called when buffers are filled up (packing) or needs refill (unpack). Containers (arrays, maps) are read/written in parts, first the item containing the size and then the contained items one by one. Exception to this is the cw_skip_items function which skip whole containers. EXAMPLE Pack and unpack example from the MessagePack home page: void example (void) { cw_pack_context pc; char buffer; cw_pack_context_init (&pc, buffer, 20, 0, 0); cw_pack_map_size (&pc, 2); cw_pack_str (&pc, "compact", 7); cw_pack_boolean (&pc, true); cw_pack_str (&pc, "schema", 6); cw_pack_unsigned (&pc, 0); int length = pc.current - pc.start; if (length > 18) ERROR; cw_unpack_context uc; cw_unpack_context_init (&uc, pc.start, length, 0, 0); cw_unpack_next(&uc); if (uc.item.type != CWP_ITEM_MAP || uc.item.as.map.size != 2) ERROR; cw_unpack_next(&uc); if (uc.item.type != CWP_ITEM_STR || uc.item.as.str.length != 7)) ERROR; if (strncmp("compact", uc.item.as.str.start, 7)) ERROR; cw_unpack_next(&uc); if (uc.item.type != CWP_ITEM_BOOLEAN || uc.item.as.boolean != true) ERROR; cw_unpack_next(&uc); if (uc.item.type != CWP_ITEM_STR || uc.item.as.str.length != 6)) ERROR; if (strncmp("schema", uc.item.as.str.start, 6)) ERROR; cw_unpack_next(&uc); if (uc.item.type != CWP_ITEM_POSITIVE_INTEGER || uc.item.as.u64 != 0) ERROR; cw_unpack_next(&uc); if (uc.return_code != CWP_RC_END_OF_INPUT) ERROR; } In the examples folder there are more examples. BACKWARD COMPATIBILITY CWPack may be run in compatibility mode. It affects only packing; EXT is considered illegal, BIN are transformed to STR and generation of STR8 is supressed. ERROR HANDLING When an error is detected in a context, the context is stopped and all future calls to that context are immediatly returned without any actions. CWPack does not check for illegal values (e.g. in STR for illegal unicode characters). BUILD CWPack consists of a single src file and two header files. It is written in strict ansi C and the files are together ~ 1.2K lines. No separate build is neccesary, just include the files in your own build. CWPack has no dependencies to other libraries. TEST Included in the test folder are a module test and a performance test and shell scripts to run them. NEUECC/MESSAGEPACK-CSHARP HTTPS://GITHUB.COM/NEUECC/MESSAGEPACK-CSHARP MESSAGEPACK FOR C# (.NET, .NET CORE, UNITY, XAMARIN) The extremely fast MessagePack serializer for C#. It is 10x faster than MsgPack-Cli and outperforms other C# serializers. MessagePack for C# also ships with built-in support for LZ4 compression - an extremely fast compression algorithm. Performance is important, particularly in applications like game development, distributed computing, microservice architecture, and caching. MessagePack has compact binary size and full set of general purpose expression. Please see the comparison with JSON, protobuf, ZeroFormatter section . If you want to know why MessagePack C# is fastest, please see performance section . INSTALLATION The library provides in NuGet except for Unity. Standard library for .NET Framework 4.5, .NET Standard 1.6 and .NET Standard 2.0(.NET Core, Xamarin). Install-Package MessagePack Visual Studio Analyzer to help object definition Install-Package MessagePackAnalyzer Extension Packages(info is see extensions section ). Install-Package MessagePack.ImmutableCollection Install-Package MessagePack.ReactiveProperty Install-Package MessagePack.UnityShims Install-Package MessagePack.AspNetCoreMvcFormatter for Unity, download from releases page, providing .unitypackage. Unity IL2CPP or Xamarin AOT Environment, check the pre-code generation section . QUICK START Define class and mark as and public members(property or field) mark as , call MessagePackSerializer.Serialize/Deserialize. ToJson helps dump binary. // mark MessagePackObjectAttribute public class MyClass { // Key is serialization index, it is important for versioning. public int Age { get; set; } public string FirstName { get; set; } public string LastName { get; set; } // public members and does not serialize target, mark IgnoreMemberttribute public string FullName { get { return FirstName + LastName; } } } class Program { static void Main(string args) { var mc = new MyClass { Age = 99, FirstName = "hoge", LastName = "huga", }; // call Serialize/Deserialize, that's all. var bytes = MessagePackSerializer.Serialize(mc); var mc2 = MessagePackSerializer.Deserialize(bytes); // you can dump msgpack binary to human readable json. // In default, MeesagePack for C# reduce property name information. // var json = MessagePackSerializer.ToJson(bytes); Console.WriteLine(json); } } By default the attribute is required. Optionally it can be unnecessary, see Object Serialization section and Formatter Resolver section for details. ANALYZER MessagePackAnalyzer helps object definition. Attributes, accessibility etc are detected and it becomes a compiler error. If you want to allow a specific type (for example, when registering a custom type), put MessagePackAnalyzer.json at the project root and make the Build Action to AdditionalFiles. This is a sample of the contents of MessagePackAnalyzer.json. BUILT-IN SUPPORT TYPES These types can serialize by default. Primitives(int, string, etc...), Enum, Nullable<>, TimeSpan, DateTime, DateTimeOffset, Nil, Guid, Uri, Version, StringBuilder, BitArray, ArraySegment<>, BigInteger, Complext, Task, Array, Array, Array, Array, KeyValuePair<,>, Tuple<,...>, ValueTuple<,...>, List<>, LinkedList<>, Queue<>, Stack<>, HashSet<>, ReadOnlyCollection<>, IList<>, ICollection<>, IEnumerable<>, Dictionary<,>, IDictionary<,>, SortedDictionary<,>, SortedList<,>, ILookup<,>, IGrouping<,>, ObservableCollection<>, ReadOnlyOnservableCollection<>, IReadOnlyList<>, IReadOnlyCollection<>, ISet<>, ConcurrentBag<>, ConcurrentQueue<>, ConcurrentStack<>, ReadOnlyDictionary<,>, IReadOnlyDictionary<,>, ConcurrentDictionary<,>, Lazy<>, Task<>, custom inherited ICollection<> or IDictionary<,> with paramterless constructor, IList, IDictionary and custom inherited ICollection or IDictionary with paramterless constructor(includes ArrayList and Hashtable). You can add custom type support and has some official/third-party extension package. for ImmutableCollections(ImmutableList<>, etc), for ReactiveProperty and for Unity(Vector3, Quaternion, etc...), for F#(Record, FsList, Discriminated Unions, etc...). Please see extensions section . MessagePack.Nil is built-in null/void/unit representation type of MessagePack for C#. OBJECT SERIALIZATION MessagePack for C# can serialze your own public Class or Struct. Serialization target must marks and . Key type can choose int or string. If key type is int, serialized format is used array. If key type is string, serialized format is used map. If you define , does not require KeyAttribute. public class Sample1 { public int Foo { get; set; } public int Bar { get; set; } } public class Sample2 { public int Foo { get; set; } public int Bar { get; set; } } public class Sample3 { // no needs KeyAttribute public int Foo { get; set; } // If ignore public member, you can use IgnoreMemberAttribute public int Bar { get; set; } } // Console.WriteLine(MessagePackSerializer.ToJson(new Sample1 { Foo = 10, Bar = 20 })); // {"foo":10,"bar":20} Console.WriteLine(MessagePackSerializer.ToJson(new Sample2 { Foo = 10, Bar = 20 })); // {"Foo":10} Console.WriteLine(MessagePackSerializer.ToJson(new Sample3 { Foo = 10, Bar = 20 })); All patterns serialization target are public instance member(field or property). If you want to avoid serialization target, you can add to target member. > target class must be public, does not allows private, internal > class. Which should uses int key or string key? I recommend use int key because faster and compact than string key. But string key has key name information, it is useful for debugging. MessagePackSerializer requests target must put attribute is for robustness. If class is grown, you need to be conscious of versioning. MessagePackSerializer uses default value if key does not exists. If uses int key, should be start from 0 and should be sequential. If unnecessary properties come out, please make a missing number. Reuse is bad. Also, if Int Key's jump number is too large, it affects binary size. public class IntKeySample { public int A { get; set; } public int B { get; set; } } // Console.WriteLine(MessagePackSerializer.ToJson(new IntKeySample())); I want to use like JSON.NET! I don't want to put attribute! If you think that way, you can use a contractless resolver. public class ContractlessSample { public int MyProperty1 { get; set; } public int MyProperty2 { get; set; } } var data = new ContractlessSample { MyProperty1 = 99, MyProperty2 = 9999 }; var bin = MessagePackSerializer.Serialize(data, MessagePack.Resolvers.ContractlessStandardResolver.Instance); // {"MyProperty1":99,"MyProperty2":9999} Console.WriteLine(MessagePackSerializer.ToJson(bin)); // You can set ContractlessStandardResolver as default. MessagePackSerializer.SetDefaultResolver(MessagePack.Resolvers.ContractlessStandardResolver.Instance); // serializable. var bin2 = MessagePackSerializer.Serialize(data); I want to serialize private member! In default, can not serialize/deserialize private members. But you can use allow-private resolver. public class PrivateSample { int x; public void SetX(int v) { x = v; } public int GetX() { return x; } } var data = new PrivateSample(); data.SetX(9999); // You can choose StandardResolverAllowPrivate or ContractlessStandardResolverAllowPrivate var bin = MessagePackSerializer.Serialize(data, MessagePack.Resolvers.DynamicObjectResolverAllowPrivate.Instance); I don't need type, I want to use like BinaryFormatter! You can use as typeless resolver and helpers. Please see Typeless section . Resolver is key customize point of MessagePack for C#. Details, please see extension point . DATACONTRACT COMPATIBILITY You can use instead of . If type is marked DataContract, you can use instead of and instead of . is same as , is same as . If use , same as . Using DataContract makes it a shared class library and you do not have to refer to MessagePack for C#. However, it is not included in analysis by Analyzer or code generation by mpc.exe. Also, functions like UnionAttribute, MessagePackFormatterAttribute, SerializationConstructorAttribute etc can not be used. For this reason, I recommend that you use the MessagePack for C# attribute basically. SERIALIZE IMMUTABLEOBJECT(SERIALIZATIONCONSTRUCTOR) MessagePack for C# supports deserialize immutable object. For example, this struct can serialize/deserialize naturally. public struct Point { public readonly int X; public readonly int Y; public Point(int x, int y) { this.X = x; this.Y = y; } } var data = new Point(99, 9999); var bin = MessagePackSerializer.Serialize(data); // Okay to deserialize immutable obejct var point = MessagePackSerializer.Deserialize(bin); MessagePackSerializer choose constructor with the least matched argument, match index if key in integer or match name(ignore case) if key is string. If encounts MessagePackDynamicObjectResolverException: can't find matched constructor parameter you should check about this. If can not match automatically, you can specify to use constructor manually by . public struct Point { public readonly int X; public readonly int Y; // If not marked attribute, used this(least matched argument) public Point(int x) { X = x; } public Point(int x, int y) { this.X = x; this.Y = y; } } SERIALIZATION CALLBACK If object implements IMessagePackSerializationCallbackReceiver, received OnBeforeSerialize and OnAfterDeserialize on serilization process. public class SampleCallback : IMessagePackSerializationCallbackReceiver { public int Key { get; set; } public void OnBeforeSerialize() { Console.WriteLine("OnBefore"); } public void OnAfterDeserialize() { Console.WriteLine("OnAfter"); } } UNION MessagePack for C# supports serialize interface. It is like XmlInclude or ProtoInclude. MessagePack for C# there called _Union_. UnionAttribute can only attach to interface or abstract class. It requires discriminated integer key and sub-type. // mark inheritance types public interface IUnionSample { } public class FooClass : IUnionSample { public int XYZ { get; set; } } public class BarClass : IUnionSample { public string OPQ { get; set; } } // --- IUnionSample data = new FooClass() { XYZ = 999 }; // serialize interface. var bin = MessagePackSerializer.Serialize(data); // deserialize interface. var reData = MessagePackSerializer.Deserialize(bin); // use type-switch of C# 7.0 switch (reData) { case FooClass x: Console.WriteLine(x.XYZ); break; case BarClass x: Console.WriteLine(x.OPQ); break; default: break; } C# 7.0 type-switch is best match for Union. Union is serialized to two-length array. IUnionSample data = new BarClass { OPQ = "FooBar" }; var bin = MessagePackSerializer.Serialize(data); // Union is serialized to two-length array, // Console.WriteLine(MessagePackSerializer.ToJson(bin)); Using Union in Abstract Class, you can use same of interface. public abstract class ParentUnionType { public int MyProperty { get; set; } } public class SubUnionType1 : ParentUnionType { public int MyProperty1 { get; set; } } public class SubUnionType2 : ParentUnionType { public int MyProperty2 { get; set; } } Serialization of inherited type, flatten in array(or map), be carefult to integer key, it cannot duplicate parent and all childrens. DYNAMIC(UNTYPED) DESERIALIZATION If use MessagePackSerializer.Deserialize or MessagePackSerializer.Deserialize, convert messagepack binary to primitive values that convert from msgpack-primitive to bool, char, sbyte, byte, short, int, long, ushort, uint, ulong, float, double, DateTime, string, byte, object, IDictionary. // sample binary. var model = new DynamicModel { Name = "foobar", Items = new { 1, 10, 100, 1000 } }; var bin = MessagePackSerializer.Serialize(model, ContractlessStandardResolver.Instance); // dynamic, untyped var dynamicModel = MessagePackSerializer.Deserialize(bin, ContractlessStandardResolver.Instance); Console.WriteLine(dynamicModel); // foobar Console.WriteLine(dynamicModel); // 100 So you can access indexer for msgpack map and array. OBJECT TYPE SERIALIZATION StandardResolver and ContractlessStandardResolver can serialize object type as concrete type by DynamicObjectTypeFallbackResolver. var objects = new object { 1, "aaa", new ObjectFieldType { Anything = 9999 } }; var bin = MessagePackSerializer.Serialize(objects); // Console.WriteLine(MessagePackSerializer.ToJson(bin)); // Support Anonymous Type Serialize var anonType = new { Foo = 100, Bar = "foobar" }; var bin2 = MessagePackSerializer.Serialize(anonType, MessagePack.Resolvers.ContractlessStandardResolver.Instance); // {"Foo":100,"Bar":"foobar"} Console.WriteLine(MessagePackSerializer.ToJson(bin2)); > Unity supports is limited. When deserializing, same as Dynamic(Untyped) Deserialization. TYPELESS Typeless API is like BinaryFormatter, embed type information to binary so no needs type to deserialize. object mc = new Sandbox.MyClass() { Age = 10, FirstName = "hoge", LastName = "huga" }; // serialize to typeless var bin = MessagePackSerializer.Typeless.Serialize(mc); // binary data is embeded type-assembly information. // Console.WriteLine(MessagePackSerializer.ToJson(bin)); // can deserialize to MyClass with typeless var objModel = MessagePackSerializer.Typeless.Deserialize(bin) as MyClass; Type information is serialized by mspgack ext format, typecode is 100. MessagePackSerializer.Typeless is shortcut of Serialize/Deserialize(TypelessContractlessStandardResolver.Instance). If you want to configure default typeless resolver, you can set by MessagePackSerializer.Typeless.RegisterDefaultResolver. TypelessFormatter can use standalone and combinate with existing resolvers. // replace `object` uses typeless MessagePack.Resolvers.CompositeResolver.RegisterAndSetAsDefault( new { MessagePack.Formatters.TypelessFormatter.Instance }, new { MessagePack.Resolvers.StandardResolver.Instance }); public class Foo { // use Typeless(this field only) public object Bar; } If type name was changed, can not deserialize. If you need to typename fallback, you can use TypelessFormatter.BindToType. MessagePack.Formatters.TypelessFormatter.BindToType = typeName => { if (typeName.StartsWith("SomeNamespace")) { typeName = typeName.Replace("SomeNamespace", "AnotherNamespace"); } return Type.GetType(typeName, false); }; PERFORMANCE Benchmarks comparing to other serializers run on Windows 10 Pro x64 Intel Core i7-6700K 4.00GHz, 32GB RAM. Benchmark code is here - and there version info , ZeroFormatter and FlatBuffers has infinitely fast deserializer so ignore deserialize performance. MessagePack for C# uses many techniques for improve performance. * Serializer uses only ref byte and int offset, don't use (Memory)Stream(call Stream api has overhead) * High-level API uses internal memory pool, don't allocate working memory under 64K * Don't create intermediate utility instance(XxxWriter/Reader, XxxContext, etc...) * Avoid boxing all codes, all platforms(include Unity/IL2CPP) * Getting cached generated formatter on static generic field(don't use dictinary-cache because dictionary lookup is overhead): see:Resolvers * Heavyly tuned dynamic il code generation: see:DynamicObjectTypeBuilder * Call PrimitiveAPI directly when il code generation knows target is primitive * Reduce branch of variable length format when il code generation knows target(integer/string) range * Don't use IEnumerable abstraction on iterate collection, see:CollectionFormatterBase and inherited collection formatters * Uses pre generated lookup table to reduce check messagepack type, see: MessagePackBinary * Uses optimized type key dictionary for non-generic methods, see: ThreadsafeTypeKeyHashTable * Avoid string key decode for lookup map(string key) key and uses automata based name lookup with il inlining code generation, see: AutomataDictionary * For string key encode, pre-generated member name bytes and use fixed sized binary copy in IL, see: UnsafeMemory.cs Before creating this library, I implemented a fast fast serializer with ZeroFormatter#Performance . And this is a further evolved implementation. MessagePack for C# is always fast, optimized for all types(primitive, small struct, large object, any collections). DESERIALIZE PERFORMANCE PER OPTIONS Performance varies depending on options. This is a micro benchamark with BenchmarkDotNet . Target object has 9 members(MyProperty1 ~ MyProperty9), value are zero. METHOD MEAN ERROR SCALED GEN 0 ALLOCATED IntKey 72.67 ns NA 1.00 0.0132 56 B StringKey 217.95 ns NA 3.00 0.0131 56 B Typeless_IntKey 176.71 ns NA 2.43 0.0131 56 B Typeless_StringKey 378.64 ns NA 5.21 0.0129 56 B MsgPackCliMap 1,355.26 ns NA 18.65 0.1431 608 B MsgPackCliArray 455.28 ns NA 6.26 0.0415 176 B ProtobufNet 265.85 ns NA 3.66 0.0319 136 B Hyperion 366.47 ns NA 5.04 0.0949 400 B JsonNetString 2,783.39 ns NA 38.30 0.6790 2864 B JsonNetStreamReader 3,297.90 ns NA 45.38 1.4267 6000 B JilString 553.65 ns NA 7.62 0.0362 152 B JilStreamReader 1,408.46 ns NA 19.38 0.8450 3552 B IntKey, StringKey, Typeless_IntKey, Typeless_StringKey are MessagePack for C# options. All MessagePack for C# options achive zero memory allocation on deserialization process. JsonNetString/JilString is deserialized from string. JsonNetStreamReader/JilStreamReader is deserialized from UTF8 byte with StreamReader. Deserialization is normally read from Stream. Thus, it will be restored from byte(or Stream) instead of string. MessagePack for C# IntKey is fastest. StringKey is slower than IntKey because matching from the character string is required. If IntKey, read array length, for(array length) { binary decode }. If StringKey, read map length, for(map length) { decode key, lookup by key, binary decode } so requires additional two steps(decode key and lookup by key). String key is often useful, contractless, simple replacement of JSON, interoperability with other languages, and more certain versioning. MessagePack for C# is also optimized for String Key. First of all, it do not decode UTF8 byte to String for matching with the member name, it will look up the byte as it is(avoid decode cost and extra allocation). And It will try to match each long type (per 8 character, if it is not enough, pad with 0) using automata and inline it when IL code generating. This also avoids calculating the hash code of byte, and the comparison can be made several times on a long unit. This is the sample decompile of generated deserializer code by ILSpy . If the number of nodes is large, search with a embedded binary search. Extra note, this is serialize benchmark result. METHOD MEAN ERROR SCALED GEN 0 ALLOCATED IntKey 84.11 ns NA 1.00 0.0094 40 B StringKey 126.75 ns NA 1.51 0.0341 144 B Typeless_IntKey 183.31 ns NA 2.18 0.0265 112 B Typeless_StringKey 193.95 ns NA 2.31 0.0513 216 B MsgPackCliMap 967.68 ns NA 11.51 0.1297 552 B MsgPackCliArray 284.20 ns NA 3.38 0.1006 424 B ProtobufNet 176.43 ns NA 2.10 0.0665 280 B Hyperion 280.14 ns NA 3.33 0.1674 704 B ZeroFormatter 149.95 ns NA 1.78 0.1009 424 B JsonNetString 1,432.55 ns NA 17.03 0.4616 1944 B JsonNetStreamWriter 1,775.72 ns NA 21.11 1.5526 6522 B JilString 547.51 ns NA 6.51 0.3481 1464 B JilStreamWriter 778.78 ns NA 9.26 1.4448 6066 B Of course, IntKey is fastest but StringKey also good. LZ4 COMPRESSION MessagePack is a fast and _compact_ format but it is not compression. LZ4 is extremely fast compression algorithm, with MessagePack for C# can achive extremely fast perfrormance and extremely compact binary size! MessagePack for C# has built-in LZ4 support. You can use LZ4MessagePackSerializer instead of MessagePackSerializer. Builtin support is special, I've created serialize-compression pipeline and special tuned for the pipeline so share the working memory, don't allocate, don't resize until finished. Serialized binary is not simply compressed lz4 binary. Serialized binary is valid MessagePack binary used ext-format and custom typecode(99). var array= Enumerable.Range(1, 100).Select(x => new MyClass { Age = 5, FirstName = "foo", LastName = "bar" }).ToArray(); // call LZ4MessagePackSerializer instead of MessagePackSerializer, api is completely same var lz4Bytes = LZ4MessagePackSerializer.Serialize(array); var mc2 = LZ4MessagePackSerializer.Deserialize(lz4Bytes); // you can dump lz4 message pack // var json = LZ4MessagePackSerializer.ToJson(lz4Bytes); Console.WriteLine(json); // lz4Bytes is valid MessagePack, it is using ext-format( ) // var rawJson = MessagePackSerializer.ToJson(lz4Bytes); Console.WriteLine(rawJson); built-in LZ4 support uses primitive LZ4 API. The LZ4 API is more efficient if you know the size of original source length. Therefore, size is written on the top. > Compression speed is not always fast. Depending on the target > binary, it may be short or longer. However, even at worst, it is > about twice, but it is still often faster than other uncompressed > serializers. If target binary size under 64 bytes, LZ4MessagePackSerializer does not compress to optimize small size serialization. COMPARE WITH PROTOBUF, JSON, ZEROFORMATTER protbuf-net is major, most used binary-format library on .NET. I love protobuf-net and respect that great work. But if uses protobuf-net for general-purpose serialization format, you may encounts annoying issue. public class Parent { public int Primitive { get; set; } public Child Prop { get; set; } public int Array { get; set; } } public class Child { public int Number { get; set; } } using (var ms = new MemoryStream()) { // serialize null. ProtoBuf.Serializer.Serialize(ms, null); ms.Position = 0; var result = ProtoBuf.Serializer.Deserialize(ms); Console.WriteLine(result != null); // True, not null. but all property are zero formatted. Console.WriteLine(result.Primitive); // 0 Console.WriteLine(result.Prop); // null Console.WriteLine(result.Array); // null } using (var ms = new MemoryStream()) { // serialize empty array. ProtoBuf.Serializer.Serialize(ms, new Parent { Array = new int }); ms.Position = 0; var result = ProtoBuf.Serializer.Deserialize(ms); Console.WriteLine(result.Array == null); // True, null! } protobuf(-net) can not handle null and empty collection correctly. Because protobuf has no null representation( this is the protobuf-net authors answer). MessagePack specification can completely serialize C# type system. This is the reason to recommend MessagePack over protobuf. Protocol Buffers has good IDL and gRPC , that is a much good point than MessagePack. If you want to use IDL, I recommend Google.Protobuf than MessagePack. JSON is good general-purpose format. It is perfect, simple and enough spec. Utf8Json which created me that adopts same architecture as MessagePack for C# and avoid encoding/decoing cost so work like binary. If you want to know about binary vs text, see Utf8Json/which serializer should be used section. ZeroFormatter is similar as FlatBuffers but specialized to C#. It is special. Deserialization is infinitely fast but instead the binary size is large. And ZeroFormatter's caching algorithm requires additional memory. Again, ZeroFormatter is special. When situation matches with ZeroFormatter, it demonstrates power of format. But for many common uses, MessagePack for C# would be better. EXTENSIONS MessagePack for C# has extension point and you can add external type's serialization support. There are official extension support. Install-Package MessagePack.ImmutableCollection Install-Package MessagePack.ReactiveProperty Install-Package MessagePack.UnityShims Install-Package MessagePack.AspNetCoreMvcFormatter MessagePack.ImmutableCollection package add support for System.Collections.Immutable library. It adds ImmutableArray<>, ImmutableList<>, ImmutableDictionary<,>, ImmutableHashSet<>, ImmutableSortedDictionary<,>, ImmutableSortedSet<>, ImmutableQueue<>, ImmutableStack<>, IImmutableList<>, IImmutableDictionary<,>, IImmutableQueue<>, IImmutableSet<>, IImmutableStack<> serialization support. MessagePack.ReactiveProperty package add support for ReactiveProperty library. It adds ReactiveProperty<>, IReactiveProperty<>, IReadOnlyReactiveProperty<>, ReactiveCollection<>, Unit serialization support. It is useful for save viewmodel state. MessagePack.UnityShims package provides shim of Unity 's standard struct(Vector2, Vector3, Vector4, Quaternion, Color, Bounds, Rect, AnimationCurve, Keyframe, Matrix4x4, Gradient, Color32, RectOffset, LayerMask, Vector2Int, Vector3Int, RangeInt, RectInt, BoundsInt) and there formatter. It can enable to commnicate between server and Unity client. After install, extension package must enable by configuration. Here is sample of enable all extension. // set extensions to default resolver. MessagePack.Resolvers.CompositeResolver.RegisterAndSetAsDefault( // enable extension packages first ImmutableCollectionResolver.Instance, ReactivePropertyResolver.Instance, MessagePack.Unity.Extension.UnityBlitResolver.Instance, MessagePack.Unity.UnityResolver.Instance, // finaly use standard(default) resolver StandardResolver.Instance); ); Configuration details, see:Extension Point section . MessagePack.AspNetCoreMvcFormatter is add-on of ASP.NET Core MVC 's serialization to boostup performance. This is configuration sample. public void ConfigureServices(IServiceCollection services) { services.AddMvc().AddMvcOptions(option => { option.OutputFormatters.Clear(); option.OutputFormatters.Add(new MessagePackOutputFormatter(ContractlessStandardResolver.Instance)); option.InputFormatters.Clear(); option.InputFormatters.Add(new MessagePackInputFormatter(ContractlessStandardResolver.Instance)); }); } Author is creating other extension packages, too. * MasterMemory - Embedded Readonly In-Memory Document Database * MagicOnion - gRPC based HTTP/2 RPC Streaming Framework * DatadogSharp - C# Datadog client You can make your own extension serializers or integrate with framework, let's create them and share it! * MessagePack.FSharpExtensions - supports F# list,set,map,unit,option,discriminated union * MessagePack.NodaTime - Support for NodaTime types to MessagePack C# * WebApiContrib.Core.Formatter.MessagePack - supports ASP.NET Core MVC(details in blog post ) * MessagePack.MediaTypeFormatter - MessagePack MediaTypeFormatter HIGH-LEVEL API(MESSAGEPACKSERIALIZER) MessagePackSerializer is the entry point of MessagePack for C#. Its static methods are main API of MessagePack for C#. API DESCRIPTION DefaultResolver FormatterResolver that used resolver less overloads. If does not set it, used StandardResolver. SetDefaultResolver Set default resolver of MessagePackSerializer APIs. Serialize Convert object to byte or write to stream. There has IFormatterResolver overload, used specified resolver. SerializeUnsafe Same as Serialize but return ArraySegement. The result of ArraySegment is contains internal buffer pool, it can not share across thread and can not hold, so use quickly. Deserialize Convert byte or ArraySegment or stream to object. There has IFormatterResolver overload, used specified resolver. NonGeneric.* NonGeneric APIs of Serialize/Deserialize. There accept type parameter at first argument. This API is bit slower than generic API but useful for framework integration such as ASP.NET formatter. Typeless.* Typeless APIs of Serialize/Deserialize. This API no needs type parameter like BinaryFormatter. This API makes .NET specific binary and bit slower than standard APIs. ToJson Dump message-pack binary to JSON string. It is useful for debugging. FromJson From Json string to MessagePack binary. ToLZ4Binary (LZ4 only)Convert msgpack binary to LZ4 msgpack binary. Decode (LZ4 only)Convert LZ4 msgpack binary to standard msgpack binary. MessagePack for C# operates at the byte level, so byte API is faster than Stream API. If byte can be used for I/O, I recommend using the byte API. Deserialize(Stream) has bool readStrict overload. It means read byte from stream strictly size. The default is false, it reads all stream data, it is faster than readStrict but if the data is contiguous, you can use readStrict = true. High-Level API uses memory pool internaly to avoid unnecessary memory allocation. If result size is under 64K, allocates GC memory only for the return bytes. LZ4MessagePackSerializer has same api with MessagePackSerializer and DefaultResolver is shared. LZ4MessagePackSerializer has additional SerializeToBlock method. LOW-LEVEL API(IMESSAGEPACKFORMATTER) IMessagePackFormatter is serializer by each type. For example Int32Formatter : IMessagePackFormatter represents Int32 MessagePack serializer. public interface IMessagePackFormatter { int Serialize(ref byte bytes, int offset, T value, IFormatterResolver formatterResolver); T Deserialize(byte bytes, int offset, IFormatterResolver formatterResolver, out int readSize); } All api works on byte level, no use Stream, no use Writer/Reader so improve performance. Many builtin formatters exists under MessagePack.Formatters. You can get sub type serializer by formatterResolver.GetFormatter. Here is sample of write own formatter. // serialize fileinfo as string fullpath. public class FileInfoFormatter : IMessagePackFormatter { public int Serialize(ref byte bytes, int offset, FileInfo value, IFormatterResolver formatterResolver) { if (value == null) { return MessagePackBinary.WriteNil(ref bytes, offset); } return MessagePackBinary.WriteString(ref bytes, offset, value.FullName); } public FileInfo Deserialize(byte bytes, int offset, IFormatterResolver formatterResolver, out int readSize) { if (MessagePackBinary.IsNil(bytes, offset)) { readSize = 1; return null; } var path = MessagePackBinary.ReadString(bytes, offset, out readSize); return new FileInfo(path); } } Created formatter needs to register to IFormatterResolver. Please see Extension Point section . You can see many other samples from builtin formatters . PRIMITIVE API(MESSAGEPACKBINARY) MessagePackBinary is most low-level API like Reader/Writer of other serializers. MessagePackBinary is static class because avoid create Reader/Writer allocation. METHOD DESCRIPTION ReadNext Skip MessagePackFormat binary block, returns read size. ReadNextBlock Skip MessagePackFormat binary block with sub structures(array/map), returns read size. This is useful for create deserializer. ReadMessageBlockFromStreamUnsafe Read binary block from Stream, if readOnlySingleMessage = false then read sub structures(array/map). ReadStringSegment Read string format but do not decode UTF8, returns ArraySegment. ReadBytesSegment Read binary format but do not copy bytes, returns ArraySegment. Write/ReadMapHeader Write/Read map format header(element length). WriteMapHeaderForceMap32Block Write map format header, always use map32 format(length is fixed, 5). Write/ReadArrayHeader Write/Read array format header(element length). WriteArrayHeaderForceArray32Block Write array format header, always use array32 format(length is fixed, 5). Write/Read*** *** is primitive type name(Int32, Single, String, etc...) Write***Force***Block *** is primitive integer name(Byte, Int32, UInt64, etc...), acquire strict block and write code Write/ReadBytes Write/Read byte to use bin format. Write/ReadExtensionFormat Write/Read ext format header(Length + TypeCode) and content byte. Write/ReadExtensionFormatHeader Write/Read ext format, header(Length + TypeCode) only. WriteExtensionFormatHeaderForceExt32Block Write ext format header, always use ext32 format(length is fixed, 6). WriteRaw Write msgpack block directly. IsNil Is TypeCode Nil? GetMessagePackType Return MessagePackType of target MessagePack bianary position. GetExtensionFormatHeaderLength Calculate extension formatter header length. GetEncodedStringBytes Get msgpack packed raw binary. EnsureCapacity Resize if byte can not fill. FastResize Buffer.BlockCopy version of Array.Resize. FastCloneWithResize Same as FastResize but return copied byte. Read API returns deserialized primitive and read size. Write API returns write size and guranteed auto ensure ref byte. Write/Read API has byte overload and Stream overload, basically the byte API is faster. DateTime is serialized to MessagePack Timestamp format , it serialize/deserialize UTC and loses Kind info. If you useNativeDateTimeResolver serialized native DateTime binary format and it can keep Kind info but cannot communicate other platforms. MessagePackType means msgpack spec of source types . MessagePackCode means msgpack format of first byte . Its static class has ToMessagePackType and ToFormatName utility methods. MessagePackRange means Min-Max fix range of msgpack format. EXTENSION POINT(IFORMATTERRESOLVER) IFormatterResolver is storage of typed serializers. Serializer api accepts resolver and can customize serialization. RESOVLER NAME DESCRIPTION BuiltinResolver Builtin primitive and standard classes resolver. It includes primitive(int, bool, string...) and there nullable, array and list. and some extra builtin types(Guid, Uri, BigInteger, etc...). StandardResolver Composited resolver. It resolves in the following order builtin -> attribute -> dynamic enum -> dynamic generic -> dynamic union -> dynamic object -> dynamic object fallback. This is the default of MessagePackSerializer. ContractlessStandardResolver Composited StandardResolver(except dynamic object fallback) -> DynamicContractlessObjectResolver -> DynamicObjectTypeFallbackResolver. It enables contractless serialization. StandardResolverAllowPrivate Same as StandardResolver but allow serialize/deserialize private members. ContractlessStandardResolverAllowPrivate Same as ContractlessStandardResolver but allow serialize/deserialize private members. PrimitiveObjectResolver MessagePack primitive object resolver. It is used fallback in object type and supports bool, char, sbyte, byte, short, int, long, ushort, uint, ulong, float, double, DateTime, string, byte, ICollection, IDictionary. DynamicObjectTypeFallbackResolver Serialize is used type in from object type, deserialize is used PrimitiveObjectResolver. AttributeFormatterResolver Get formatter from attribute. CompositeResolver Singleton helper of setup custom resolvers. You can use Register or RegisterAndSetAsDefault API. NativeDateTimeResolver Serialize by .NET native DateTime binary format. UnsafeBinaryResolver Guid and Decimal serialize by binary representation. It is faster than standard(string) representation. OldSpecResolver str and bin serialize/deserialize follows old messagepack spec(use raw format) DynamicEnumResolver Resolver of enum and there nullable, serialize there underlying type. It uses dynamic code generation to avoid boxing and boostup performance serialize there name. DynamicEnumAsStringResolver Resolver of enum and there nullable. It uses reflection call for resolve nullable at first time. DynamicGenericResolver Resolver of generic type(Tuple<>, List<>, Dictionary<,>, Array, etc). It uses reflection call for resolve generic argument at first time. DynamicUnionResolver Resolver of interface marked by UnionAttribute. It uses dynamic code generation to create dynamic formatter. DynamicObjectResolver Resolver of class and struct maked by MessagePackObjectAttribute. It uses dynamic code generation to create dynamic formatter. DynamicContractlessObjectResolver Resolver of all classes and structs. It does not needs MessagePackObjectAttribute and serialized key as string(same as marked ). DynamicObjectResolverAllowPrivate Same as DynamicObjectResolver but allow serialize/deserialize private members. DynamicContractlessObjectResolverAllowPrivate Same as DynamicContractlessObjectResolver but allow serialize/deserialize private members. TypelessObjectResolver Used for object, embed .NET type in binary by ext(100) format so no need to pass type in deserilization. TypelessContractlessStandardResolver Composited resolver. It resolves in the following order nativedatetime -> builtin -> attribute -> dynamic enum -> dynamic generic -> dynamic union -> dynamic object -> dynamiccontractless -> typeless. This is the default of MessagePackSerializer.Typeless It is the only configuration point to assemble the resolver's priority. In most cases, it is sufficient to have one custom resolver globally. CompositeResolver will be its helper. // use global-singleton CompositeResolver. // This method initialize CompositeResolver and set to default MessagePackSerializer CompositeResolver.RegisterAndSetAsDefault( // resolver custom types first ImmutableCollectionResolver.Instance, ReactivePropertyResolver.Instance, MessagePack.Unity.Extension.UnityBlitResolver.Instance, MessagePack.Unity.UnityResolver.Instance, // finaly use standard resolver StandardResolver.Instance); Here is sample of use DynamicEnumAsStringResolver with DynamicContractlessObjectResolver(It is JSON.NET-like lightweight setting.) // composite same as StandardResolver CompositeResolver.RegisterAndSetAsDefault( MessagePack.Resolvers.BuiltinResolver.Instance, MessagePack.Resolvers.AttributeFormatterResolver.Instance, // replace enum resolver MessagePack.Resolvers.DynamicEnumAsStringResolver.Instance, MessagePack.Resolvers.DynamicGenericResolver.Instance, MessagePack.Resolvers.DynamicUnionResolver.Instance, MessagePack.Resolvers.DynamicObjectResolver.Instance, MessagePack.Resolvers.PrimitiveObjectResolver.Instance, // final fallback(last priority) MessagePack.Resolvers.DynamicContractlessObjectResolver.Instance ); If you want to write custom composite resolver, you can write like following. public class CustomCompositeResolver : IFormatterResolver { public static IFormatterResolver Instance = new CustomCompositeResolver(); static readonly IFormatterResolver resolvers = new { // resolver custom types first ImmutableCollectionResolver.Instance, ReactivePropertyResolver.Instance, MessagePack.Unity.Extension.UnityBlitResolver.Instance, MessagePack.Unity.UnityResolver.Instance, // finaly use standard resolver StandardResolver.Instance }; CustomCompositeResolver() { } public IMessagePackFormatter GetFormatter() { return FormatterCache.formatter; } static class FormatterCache { public static readonly IMessagePackFormatter formatter; static FormatterCache() { foreach (var item in resolvers) { var f = item.GetFormatter(); if (f != null) { formatter = f; return; } } } } } IFormatterResolver can use per serialize/deserialize method. MessagePackSerializer.Serialize(data, CustomCompositeResolver.Instance); If you want to make your extension package, you need to make formatter and resolver. IMessagePackFormatter accepts IFormatterResolver on every request of serialize/deserialize. You can get child-type serialize on resolver.GetFormatterWithVerify. Here is sample of own resolver. public class SampleCustomResolver : IFormatterResolver { // Resolver should be singleton. public static IFormatterResolver Instance = new SampleCustomResolver(); SampleCustomResolver() { } // GetFormatter's get cost should be minimized so use type cache. public IMessagePackFormatter GetFormatter() { return FormatterCache.formatter; } static class FormatterCache { public static readonly IMessagePackFormatter formatter; // generic's static constructor should be minimized for reduce type generation size! // use outer helper method. static FormatterCache() { formatter = (IMessagePackFormatter)SampleCustomResolverGetFormatterHelper.GetFormatter(typeof(T)); } } } internal static class SampleCustomResolverGetFormatterHelper { // If type is concrete type, use type-formatter map static readonly Dictionary formatterMap = new Dictionary() { {typeof(FileInfo), new FileInfoFormatter()} // add more your own custom serializers. }; internal static object GetFormatter(Type t) { object formatter; if (formatterMap.TryGetValue(t, out formatter)) { return formatter; } // If target type is generics, use MakeGenericType. if (t.IsGenericParameter && t.GetGenericTypeDefinition() == typeof(ValueTuple<,>)) { return Activator.CreateInstance(typeof(ValueTupleFormatter<,>).MakeGenericType(t.GenericTypeArguments)); } // If type can not get, must return null for fallback mecanism. return null; } } MESSAGEPACKFORMATTERATTRIBUTE MessagePackFormatterAttribute is lightweight extension point of class, struct, interface, enum and property/field. This is like JSON.NET's JsonConverterAttribute. For example, serialize private field, serialize x10 formatter. public class CustomObject { string internalId; public CustomObject() { this.internalId = Guid.NewGuid().ToString(); } // serialize/deserialize internal field. class CustomObjectFormatter : IMessagePackFormatter { public int Serialize(ref byte bytes, int offset, CustomObject value, IFormatterResolver formatterResolver) { return formatterResolver.GetFormatterWithVerify().Serialize(ref bytes, offset, value.internalId, formatterResolver); } public CustomObject Deserialize(byte bytes, int offset, IFormatterResolver formatterResolver, out int readSize) { var id = formatterResolver.GetFormatterWithVerify().Deserialize(bytes, offset, formatterResolver, out readSize); return new CustomObject { internalId = id }; } } } // per field, member public class Int_x10Formatter : IMessagePackFormatter { public int Deserialize(byte bytes, int offset, IFormatterResolver formatterResolver, out int readSize) { return MessagePackBinary.ReadInt32(bytes, offset, out readSize) * 10; } public int Serialize(ref byte bytes, int offset, int value, IFormatterResolver formatterResolver) { return MessagePackBinary.WriteInt32(ref bytes, offset, value * 10); } } public class MyClass { // You can attach custom formatter per member. public int MyProperty1 { get; set; } } Formatter is retrieved by AttributeFormatterResolver, it is included in StandardResolver. IGNOREFORMATTER IgnoreFormatter is lightweight extension point of class and struct, if exists can't serializable type in external type, you can register IgnoreFormatter that serialize to nil. // CompositeResolver can set custom formatter. MessagePack.Resolvers.CompositeResolver.RegisterAndSetAsDefault( new IMessagePackFormatter { // for example, register reflection infos(can not serialize in default) new IgnoreFormatter(), new IgnoreFormatter(), new IgnoreFormatter(), new IgnoreFormatter() }, new IFormatterResolver { ContractlessStandardResolver.Instance }); RESERVED EXTENSION TYPES MessagePack for C# already used some messagepack ext type codes, be careful to use same ext code. CODE TYPE USE BY -1 DateTime msgpack-spec reserved for timestamp 30 Vector2 for Unity, UnsafeBlitFormatter 31 Vector3 for Unity, UnsafeBlitFormatter 32 Vector4 for Unity, UnsafeBlitFormatter 33 Quaternion for Unity, UnsafeBlitFormatter 34 Color for Unity, UnsafeBlitFormatter 35 Bounds for Unity, UnsafeBlitFormatter 36 Rect for Unity, UnsafeBlitFormatter 37 Int for Unity, UnsafeBlitFormatter 38 Float for Unity, UnsafeBlitFormatter 39 Double for Unity, UnsafeBlitFormatter 99 All LZ4MessagePackSerializer 100 object TypelessFormatter FOR UNITY You can install by package and includes source code. If build target as PC, you can use as is but if build target uses IL2CPP, you can not use Dynamic***Resolver so use pre-code generation. Please see pre-code generation section . In Unity, MessagePackSerializer can serialize Vector2, Vector3, Vector4, Quaternion, Color, Bounds, Rect, AnimationCurve, Keyframe, Matrix4x4, Gradient, Color32, RectOffset, LayerMask, Vector2Int, Vector3Int, RangeInt, RectInt, BoundsInt and there nullable, there array, there list by built-in extension UnityResolver. It is included StandardResolver by default. MessagePack for C# has additional unsafe extension. UnsafeBlitResolver is special resolver for extremely fast unsafe serialization/deserialization for struct array. x20 faster Vector3 serialization than native JsonUtility. If use UnsafeBlitResolver, serialize special format(ext:typecode 30~39) Vector2, Vector3, Quaternion, Color, Bounds, Rect. If use UnityBlitWithPrimitiveArrayResolver, supports int, float, double too. This special feature is useful for serialize Mesh(many Vector3) or many transform position. If you want to use unsafe resolver, you must enables unsafe option and define additional symbols. At first, write -unsafe on smcs.rsp, gmcs.rsp etc. And define ENABLE_UNSAFE_MSGPACK symbol. Here is sample of configuration. Resolvers.CompositeResolver.RegisterAndSetAsDefault( MessagePack.Unity.UnityResolver.Instance, MessagePack.Unity.Extension.UnityBlitWithPrimitiveArrayResolver.Instance, // If PC, use StandardResolver // MessagePack.Resolvers.StandardResolver.Instance, // If IL2CPP, Builtin + GeneratedResolver. // MessagePack.Resolvers.BuiltinResolver.Instance, ); MessagePack.UnityShims NuGet package is for .NET ServerSide serialization support to communicate with Unity. It includes shim of Vector3 etc and Safe/Unsafe serialization extension. If you want to share class between Unity and Server, you can use SharedProject or Reference as Link or new MSBuild(VS2017)'s wildcard reference etc. Anyway you need to source-code level share. This is sample project structure of use SharedProject. * SharedProject(source code sharing) * Source codes of server-client shared * ServerProject(.NET 4.6/.NET Core/.NET Standard) * * * * ClientDllProject(.NET 3.5) * * (not dll, use MessagePack.unitypackage's sourcecodes) * Unity * Other ways, use plain POCO by DataContract/DataMember can use. PRE CODE GENERATION(UNITY/XAMARIN SUPPORTS) MessagePack for C# generates object formatter dynamically by ILGenerator . It is fast and transparently generated at run time. But it needs generate cost at first time and it does not work on AOT environment(Xamarin, Unity IL2CPP, etc.). > Note: If Unity's build target as PC, does not need code generation. > It works well. If you want to avoid generate cost or run on Xamarin or Unity, you need pre-code generation. mpc.exe(MessagePackCompiler) is code generator of MessagePack for C#. mpc can download from releases page, mpc.zip. mpc is using Roslyn so analyze source code. mpc arguments help: -i, --input Input path of analyze csproj -o, --output Output file path -c, --conditionalsymbol conditional compiler symbol -r, --resolvername Set resolver name -n, --namespace Set namespace root name -m, --usemapmode Force use map mode serialization // Simple Sample: mpc.exe -i "..\src\Sandbox.Shared.csproj" -o "MessagePackGenerated.cs" // Use force map simulate DynamicContractlessObjectResolver mpc.exe -i "..\src\Sandbox.Shared.csproj" -o "MessagePackGenerated.cs" -m If you create DLL by msbuild project, you can use Pre/Post build event. Resolver at first. // CompositeResolver is singleton helper for use custom resolver. // Ofcourse you can also make custom resolver. MessagePack.Resolvers.CompositeResolver.RegisterAndSetAsDefault( // use generated resolver first, and combine many other generated/custom resolvers MessagePack.Resolvers.GeneratedResolver.Instance, // finally, use builtin/primitive resolver(don't use StandardResolver, it includes dynamic generation) MessagePack.Resolvers.BuiltinResolver.Instance, MessagePack.Resolvers.AttributeFormatterResolver.Instance, MessagePack.Resolvers.PrimitiveObjectResolver.Instance ); > Note: mpc.exe is basically run on only Windows. But you can run on > Mono , that supports Mac and Linux. RPC MessagePack advocated MessagePack RPC , but formulation is stopped and it is not widely used. I've created gRPC based MessagePack HTTP/2 RPC streaming framework called MagicOnion . gRPC usually communicates with Protocol Buffers using IDL. But MagicOnion uses MessagePack for C# and does not needs IDL. If communicates C# to C#, schemaless(C# classes as schema) is better than IDL. HOW TO BUILD Open MessagePack.sln on Visual Studio 2017. Unity Project is using symbolic link. At first, run make_unity_symlink.bat so linked under Unity project. You can open src\MessagePack.UnityClient on Unity Editor. AUTHOR INFO Yoshifumi Kawai(a.k.a. neuecc) is a software developer in Japan. He is the Director/CTO at Grani, Inc. Grani is a mobile game developer company in Japan and well known for using C#. He is awarding Microsoft MVP for Visual C# since 2011. He is known as the creator of UniRx (Reactive Extensions for Unity) Blog: https://medium.com/@neuecc (English) Blog: http://neue.cc/ (Japanese) Twitter: https://twitter.com/neuecc (Japanese) LICENSE This library is under the MIT License. lz4 compression support is using Milosz Krajewski's lz4net code with some modified. POCKETBERSERKER/MESSAGEPACK.FSHARPEXTENSIONS HTTPS://GITHUB.COM/POCKETBERSERKER/MESSAGEPACK.FSHARPEXTENSIONS MESSAGEPACK.FSHARPEXTENSIONS MessagePack.FSharpExtensions is a MessagePack-CSharp extension library for F#. USAGE open MessagePack open MessagePack.Resolvers open MessagePack.FSharp CompositeResolver.RegisterAndSetAsDefault( FSharpResolver.Instance, StandardResolver.Instance ) type UnionSample = | Foo of XYZ : int | Bar of OPQ : string list let data = Foo 999 let bin = MessagePackSerializer.Serialize(data) match MessagePackSerializer.Deserialize(bin) with | Foo x -> printfn "%d" x | Bar xs -> printfn "%A" xs JPF91/MSGPACK-LL HTTPS://GITHUB.COM/JPF91/MSGPACK-LL MSGPACK-LL This is a low-level @nogc, nothrow, @safe, pure and betterC compatible MessagePack serializer and deserializer. The library was designed to avoid any external dependencies and handle the low-level protocol details only. As a result the library doesn't have to do any error handling or buffer management. This library does never dynamically allocate memory. The API documentation is available here . A SIMPLE EXAMPLE SHOWING THE COMPLETE API import msgpack_ll; // Buffer allocation is not handled by the library ubyte buffer; // The MsgpackType enum contains all low-level MessagePack types enum type = MsgpackType.uint8; // The DataSize!(MsgpackType) function returns the size of serialized data // for a certain type. // The formatter and parser use ref ubyte types. This // forces the compiler to do array length checks at compile time and avoid // any runtime bounds checking. // Format the number 42 as a uint8 type. This will require // DataSize!(MsgpackType.uint8) == 2 bytes storage. formatType!(type)(42, buffer); // To deserialize we have to somehow get the data type at runtime // Then verify the type is as expected. assert(getType(buffer) == type); // Now deserialize. Here we have to specify the MsgpackType // as a compile time value. const result = parseType!type(buffer); assert(result == 42); A QUICK VIEW AT THE GENERATED CODE FOR THIS LIBRARY Serializing an 8 bit integer void format(ref ubyte buffer) { enum type = MsgpackType.uint8; formatType!(type)(42, buffer); } Because of clever typing there's no runtime bounds checking but all bounds checks are performed at compile time by type checking. pure nothrow @nogc @safe void msgpack_ll.format(ref ubyte): mov BYTE PTR , -52 mov BYTE PTR , 42 ret Serializing a small negative integer into one byte void format(ref ubyte buffer) { enum type = MsgpackType.negFixInt; formatType!(type)(-11, buffer); } The MessagePack format is cleverly designed, so encoding the type is actually free in this case. pure nothrow @nogc @safe void msgpack_ll.format(ref ubyte): mov BYTE PTR , -11 ret Deserializing an expected type bool parse(ref ubyte buffer, ref byte value) { enum type = MsgpackType.negFixInt; auto rtType = getType(buffer); if(rtType != type) return false; value = parseType!type(buffer); return true; } The compiler will inline functions and can see through the switch block in getType. If you explicitly ask for one type, the compiler will reduce the code to a simple explicit if check for this type! pure nothrow @nogc @safe bool msgpack_ll.parse(ref ubyte, ref byte): movzx edx, BYTE PTR cmp edx, 223 jle .L58 mov BYTE PTR , dl mov eax, 1 ret .L58: xor eax, eax ret Deserializing one of multiple types bool parse(ref ubyte buffer, ref byte value) { auto rtType = getType(buffer); switch(rtType) { case MsgpackType.negFixInt: value = parseType!(MsgpackType.negFixInt)(buffer); return true; case MsgpackType.int8: value = parseType!(MsgpackType.int8)(buffer); return true; default: return false; } } The generated code is obviously slighly more complex. The interesting part here is that type checking is directly done using the raw type value and not the enum values returned by getType. Even manually written ASM probably can't do much better here. pure nothrow @nogc @safe bool msgpack_ll.parse(ref ubyte, ref byte): movzx ecx, BYTE PTR xor eax, eax cmp ecx, 191 jle .L55 cmp ecx, 223 jg .L56 cmp ecx, 208 jne .L60 movzx eax, BYTE PTR mov BYTE PTR , al mov eax, 1 .L55: rep; ret .L56: mov BYTE PTR , cl mov eax, 1 ret .L60: ret MOBILTRONINC/MSGPACK-RESPONSE HTTPS://GITHUB.COM/MOBILTRONINC/MSGPACK-RESPONSE MSGPACK-RESPONSE An implementation of Message Pack middleware for ExpressJS . FEATURES * Automatic Message Pack detection (from the HTTP headers) and encoding of all JSON messages to Message Pack. * Extension of the current ExpressJS API; Introducing the Response.msgPack(jsObject) method on the standard ExpressJS Response object. GETTING STARTED With auto-detection and transformation enabled, the middleware detects automatically the HTTP header Accept: application/x-msgpack and piggybacks the Response.json() method of the ExpressJS API, to encode the JSON response as Message Pack. This method is usefull, when you have existing applications that need use the middleware, without changing the codebase very much. const msgpackResponse = require('msgpack-response'); app.use(msgpackResponse({auto_detect: true})); app.get('/test_json', (req, res) => { res.status(200).json({'message': 'a true test'}); }) > Note: Remember the add the header Accept: application/x-msgpack in > the request. Also it can have auto detection and transformation disabled. The middleware extends the Response object of the ExpressJS framework, by adding the msgPack() method to it. Then to return an encoded response, you just use the Response.msgPack() method that accepts the Javascript object as parameter. For example, const msgpackResponse = require('msgpack-response'); app.use(msgpackResponse({auto_detect: false})); //or app.use(msgpackResponse()); app.get('/test_msgpack', (req, res) => { res.status(200).msgPack({'message': 'a true test'}); }); > Note: Initialize the middleware before the actual routes in the > middleware chain to properly extend the Response Object. REQUIREMENTS Node.js >= 6.0 INSTALLATION With npm do: npm install msgpack-response -save ABOUT This project is funded and maintained by Mobiltron, Inc. . We ❤️ open source software! Check out our other open source projects or say 👋 on twitter @mobiltron . CONTRIBUTE Contributions are welcome 🤘 We encourage developers like you to help us improve the projects we've shared with the community. Please see the Contributing Guide and the Code of Conduct . AUTHORS * STAVROS SCHIZAS - _Initial work_ - Mobiltron, Inc. * VASSILIOS KARAKOIDAS - _Initial work_ - Mobiltron, Inc. See also the list of contributors who participated in this project. LICENSE msgpack-response is available under the MIT license. See the LICENSE file for more info. CROWDING/MSGPACK-R HTTPS://GITHUB.COM/CROWDING/MSGPACK-R MSGPACK FOR R Peter Meilstrup This is a high speed msgpack encoder and decoder for R, based on the CWPack C implementation. msgpack is a binary data format with data structures similar to JSON and a compact binary encoding. It can be a drop-in replacement for JSON in most applications. It is designed to be fast to parse and compact to transmit and store. INSTALLATION From CRAN: install.packages("msgpack") From Github: library(devtools) install_github("crowding/msgpack-r") USAGE library(msgpack) x <- packMsg( list(compact=TRUE, schema=0) ) x ## 82 a7 63 6f 6d 70 61 63 74 c3 a6 73 63 68 65 6d 61 00 dput(unpackMsg( x )) ## structure(list(compact = TRUE, schema = 0L), .Names = c("compact", ## "schema")) Connections / Streaming Write messages one or several at a time: conOut <- rawConnection(raw(0), open = "w") # or socketConnection, etc writeMsg("one", conOut) writeMsgs(list(2, c(buckle=TRUE), c(owner="my", type="shoe")), conOut) Use a msgConnection object to read messages one or several at a time: conIn <- msgConnection(rawConnection(rawConnectionValue(conOut), open = "r")) dput(readMsgs(conIn, 2)) ## list("one", 2L) dput(readMsg(conIn)) ## structure(TRUE, .Names = "buckle") dput(readMsgs(conIn)) ## list(structure(c("my", "shoe"), .Names = c("owner", "type"))) Performance Msgpack is fast and compact. See the benchmarking vignette . KOMAMITSU/RETROFIT-CONVERTER-MSGPACK HTTPS://GITHUB.COM/KOMAMITSU/RETROFIT-CONVERTER-MSGPACK RETROFIT-CONVERTER-MESSAGEPACK Retrofit Converter for MessagePack INSTALL Gradle dependencies { implementation 'org.komamitsu:retrofit-converter-msgpack:x.x.x' } Maven org.komamitsu retrofit-converter-msgpack x.x.x USAGE To use, supply an instance of this converter when building your Retrofit instance. Retrofit retrofit = new Retrofit.Builder() .baseUrl("https://api.example.com") .addConverterFactory(MessagePackConverterFactory.create()) .build(); PATRIKSIMEK/MSGPACK-POSTGRES HTTPS://GITHUB.COM/PATRIKSIMEK/MSGPACK-POSTGRES MSGPACK-POSTGRES MessagePack implementation for PostgreSQL written in PL/pgSQL. INSTALLATION Execute src/encode.sql or/and src/decode.sql on your database server. QUICK EXAMPLE select msgpack_encode('{"hello": "world"}'::jsonb); -- returns 0x81a568656c6c6fa5776f726c64 select msgpack_decode(decode('81a568656c6c6fa5776f726c64', 'hex')); -- returns '{"hello": "world"}' DOCUMENTATION msgpack_encode(jsonb) Encodes jsonb object into bytea string. msgpack_decode(bytea) Decodes jsonb object from bytea string. TODO * Float/double encoding SPONSORS Development is sponsored by Integromat . LICENSE Copyright (c) 2017 Patrik Simek The MIT License Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. DEV1AN/MSGPACK HTTPS://GITHUB.COM/DEV1AN/MSGPACK USAGE import MsgPack // Encode standard types let encoder = Encoder() try encoder.encode("Hello world") try encoder.encode("😇") try encoder.encode(0x0102030405060708) try encoder.encode() // Encode custom types with Encodable 🎉 struct Point: Encodable { let x: Int let y: Int } try encoder.encode(Point(x: 90, y: 45)) Take a look at the playground for more examples. HIPHISH/MSGPACK-RACKET HTTPS://GITLAB.COM/HIPHISH/MSGPACK.RKT MESSAGEPACK IN RACKET This is a Racket implementation of MessagePack , a binary data serialisation format. It allows you to serialise (pack) and de-serialise (unpack) Racket object to and from binary data. INSTALLATION The easiest way to install this library is from the Racket Package Catalog . Run the following code from your shell: raco pkg install msgpack If you wish to install the package from this repository use the included makefile: make install # Install the package make remove # Uninstall the package USING MESSAGEPACK ;;; Import the library first (require msgpack) ;;; Some object to pack (define hodgepodge (vector 1 2 (void) '#(3 #t) "foo")) ;;; Packing data (define packed (call-with-output-bytes (λ (out) (pack hodgepodge out)))) ;;; > #"\225\1\2\300\222\3\303\243foo" ;;; Unpacking data (define unpacked (call-with-input-bytes packed (λ (in) (unpack in)))) ;;; > '#(1 2 # #(3 #t) "foo") The pack function takes a Racket object and a binary output port as arguments and writes the serialised data to the port. The unpack function takes a binary input port and returns one de-serialised object, consuming the necessary amount of bytes from the port in the process. For more details please refer to the documentation . In the above example code we set the output and input ports to be byte strings so we could work with the packed and unpacked data directly inside the Racket instance. STATUS The library is fully functional, covered by test cases, and the API should be reasonably mature, but I am not yet willing to completely rule out changes. See also below for parts of the library that could not be tested at the moment due to technical reasons. CAVEATS The following cases cannot be tested for the time being: * The bin32 type, storing a byte string that is 2^32 bytes long requires 4GiB, my machine simply runs out of memory. * The same goes for the str32 type * The same goes for the array32 type * The same goes for the map32 type * The same goes for the ext32 type * Strings are only tested using ASCII characters, if anyone can generate UTF-8 strings with a given length in _bytes_ please help out. LICENSE Released under the GPLv3+ license, see the COPYING file for details. SKYFORMAT99/MSGPACK2R HTTPS://GITHUB.COM/SKYFORMAT99/MSGPACK2R MSGPACK2R Convert to and from msgpack objects in R using the official msgpack-c API through Rcpp. _A flowchart describing the conversion of R objects into msgpack objects and back._ Msgpack EXT types are converted to raw vectors with EXT attributes containing the extension type. The extension type must be an integer from 0 to 127. Maps are converted to data.frames with additional class "map". Map objects in R contain key and value list columns and can be simplified to named lists or named vectors. The helper function msgpack_map creates map objects that can be serialized into msgpack. For more information on msgpack types, see here . Installation: * On Windows, install Rtools (https://cran.r-project.org/bin/windows/Rtools/) * In R, install devtools: install.packages("devtools") * library(devtools) * install_github("traversc/msgpack2R") Example: See examples/tests.r for more examples. library(msgpack2R) library(microbenchmark) x <- as.list(1:1e7) microbenchmark(xpk <- msgpack_pack(x), times=3) # ~ 0.5 seconds microbenchmark(xu <- msgpack_unpack(xpk), times=3) # ~ 2.5-3 seconds SEANTALLEN/PONY-MSGPACK HTTPS://GITHUB.COM/SEANTALLEN/PONY-MSGPACK PONY-MSGPACK A pure Pony implementation of the MessagePack serialization format . STATUS Pony-msgpack is currently alpha software. It implements a low-level API for encoding and decoding data. Still to do: * High-level API for a better programming experience * Timestamp extension type INSTALLATION * Install pony-stable * Update your bundle.json { "type": "github", "repo": "SeanTAllen/pony-msgpack", "tag": "0.2.0" } * stable fetch to fetch your dependencies * use "msgpack" to include this package * stable env ponyc to compile your application FORZAFOOTBALL/MSGPACK-OBJC HTTPS://GITHUB.COM/FORZAFOOTBALL/MSGPACK-OBJC MSGPACK-OBJC An Objective-C wrapper for msgpack-c . Focuses on ease of use and speed. If you need configurability, there are other, more advanced libraries, for example MPMessagePack . This library will always try to use sane defaults. If any nil value is encountered in the MessagePack-data, the object will be omitted instead of returning an . This means that there can be no nil objects in dictionaries, and object-less keys will be lost in translation. Example usage: #import NSDictionary *dictionary = @{@"name": @"msgpack-objc"}; NSData *messagePackData = ; NSDictionary *unpackedDictionary = ; Supported native types: * NSArray * NSData * NSDate (using MessagePack timestamps ) * NSDictionary * NSNumber (boolean, u64, i64, float32/64) * NSString EXTENSION SUPPORT The library supports MessagePack timestamps , and will return an NSDate-object whenever one is encountered. When serializing, any NSDate-objects will also be serialized as native MessagePack timestamps. You can add native serialization for your own classes by conforming to protocol MessagePackSerializable and register it like this: ; LEKMA/MOOD.MSGPACK HTTPS://GITHUB.COM/LEKMA/MOOD.MSGPACK MOOD.MSGPACK Python MessagePack implementation MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON. But it's faster and smaller. Small integers are encoded into a single byte, and typical short strings require only one extra byte in addition to the strings themselves. NOTE: This implementation is not geared towards inter language exchange but towards serialization/de-serialization of Python object structures (it was designed as a pickle substitute). It does not expose MessagePack's extensions mechanism but uses it internally to pack/unpack non-standard types. That said, if you only deal with standard objects/types (None, True, False, integers, floating point numbers, bytes, strings, tuples and dictionaries) you are fine to use this module to produce or consume data that is targeted at or originates from other programming languages. The following documentation is largely adapted from Python's pickle module documentation . SEE ALSO: MessagePack specification ------------------------- WHAT CAN BE PACKED AND UNPACKED? The following standard types can be packed/unpacked: * None, True and False * integers, floating point numbers * strings, bytes * tuples and dictionaries containing only packable objects Additionally, the following Python types are supported by default: * complex numbers * bytearrays * lists, sets and frozensets containing only packable objects * classes (these MUST be registered in order to be unpacked) * instances of classes whose __reduce__ method conforms to the interface defined in Packing Class Instances MODULE INTERFACE * register(object) Add _object_ to the _registry_. _object_ must be a class or a singleton (instance whose __reduce__ method returns a string). * pack(object) Return the packed representation of _object_ as a bytearray object. * unpack(message) Read a packed object hierarchy from a bytes-like _message_ and return the reconstituted object hierarchy specified therein. PACKING CLASS INSTANCES NOTE: This being chiefly based on pickle's object.__reduce__() interface , most built-in objects and most objects defined in the Python standard library already conform to it. * object.__reduce__() The interface is currently defined as follows: the __reduce__ method takes no argument and shall return either a unique string or preferably a tuple. If a string is returned, the string should be interpreted as the name of a global variable. This behaviour is typically useful for singletons (and is the case for built-in functions). When a tuple is returned, it must be between two and five items long. Optional items can either be omitted, or None can be provided as their value. The semantics of each item are in order: * A callable object that will be invoked to create the initial version of the object (it MUST be registered in order to be unpacked). * A tuple of arguments for the callable object. An empty tuple must be given if the callable does not accept any argument. * Optionally, the object's state, which will be passed to the object's __setstate__ method. If the object has no such method then, the value must be a dictionary and it will be added to the object's __dict__ attribute. * Optionally, a sequence/iterator yielding successive items. These items will be appended to the object using object.extend(items). This is primarily used for list subclasses, but may be used by other classes as long as they have an extend method with the appropriate signature. If the object has no such method then, an in-place concatenation will be attempted (equivalent to object += items). * Optionally, a dict/mapping or a sequence/iterator yielding successive key-value pairs. These pairs will be stored in the object using object.update(pairs). This is primarily used for dictionary subclasses, but may be used by other classes as long as they have an update method with the appropriate signature. If the object has no such method then, an attempt will be made to store these pairs using object = value. A simple example: >>> from mood.msgpack import pack, unpack, register >>> class Kiki(object): ... def __init__(self, a, b): ... self.a = a ... self.b = b ... def __repr__(self): ... return "".format(self) ... def __reduce__(self): ... return (Kiki, (self.a, self.b)) ... >>> k = Kiki(1, 2) >>> k >>> b = pack(k) >>> b bytearray(b'\xc7\x15\x7f\x92\xc7\x0e\x06\xa8__main__\xa4Kiki\x92\x01\x02') >>> unpack(b) Traceback (most recent call last): File "", line 1, in TypeError: cannot unpack >>> register(Kiki) >>> unpack(b) >>> DCHENK/MSGP HTTPS://GITHUB.COM/DCHENK/MSGP MESSAGEPACK CODE GENERATOR AND LIBRARY FOR GO This is a code generation tool and serialization library for MessagePack. You can read more about MessagePack in the wiki or at msgpack.org . THIS LIBRARY IS NO LONGER BEING MAINTAINED. YOU SHOULD BE USING PROTOCOL BUFFERS FOR BINARY DATA SERIALIZATION Top Features * Use Go as your schema language * Performance is amazing * JSON interoperability * Type safety * Support for complex type declarations * Define your own MessagePack extensions * Automatic unit test and benchmark generation * Native support for Go’s time.Time, complex64, and complex128 types * Preprocessor directives * Generation of both byte-oriented and io.Reader/io.Writer-oriented methods Quickstart In a Go code source file, include the following directive: //go:generate msgp Within the directory with the file where you placed that directive, run go generate. The msgp command will tell the go generate tool to generate serialization and deserialization methods (functions) for all exported types in the file. These custom-built functions enable you to use package msgp to efficiently encode objects to and from the MessagePack format. You can read more about the code generation options here . Use Struct field names can be set the same way as with the encoding/json package. For example: type Person struct { Name string `msgp:"name"` Address string `msgp:"address"` Age int `msgp:"age"` Hidden string `msgp:"-"` // this field is ignored unexported bool // this field is also ignored } (The struct field tags are optional.) By default, the code generator will satisfy msgp.Sizer, msgp.Encoder, msgp.Decoder, msgp.Marshaler, and msgp.Unmarshaler. You’ll often find that much marshalling and unmarshalling will be done with zero heap allocations. Although msgp.Marshaler and msgp.Unmarshaler are similar to the standard library’s json.Marshaler and json.Unmarshaler, msgp.Encoder and msgp.Decoder are useful for stream serialization. (*msgp.Writer and *msgp.Reader are essentially protocol-aware versions of *bufio.Writer and *bufio.Reader.) Consider the following: const Eight = 8 type MyInt int type Data byte type Struct struct { Which map*MyInt `msgp:"which"` Other Data `msgp:"other"` Nums float64 `msgp:"nums"` } As long as the declarations of MyInt and Data are in the same file as Struct, the parser will determine that the type information for MyInt and Data can be passed into the definition of Struct before its methods are generated. Extensions MessagePack supports defining your own types through "extensions," which are just a tuple of the data "type" (int8) and the raw binary. You can see a worked example in the wiki. Status The code generator here and runtime library are both stable. Newer versions of the code may generate different code than older versions for performance reasons. You can read more about how msgp maps MessagePack types onto Go types in the wiki . Here some of the known limitations/restrictions: * Identifiers from outside the processed source file are assumed to satisfy the generator's interfaces. If this isn't the case, your code will fail to compile. * The chan and func fields and types are ignored as well as un-exported fields. * Encoding of interface{} is limited to built-ins or types that have explicit encoding methods. * Maps must have string keys. This is intentional (as it preserves JSON interoperability). Although non-string map keys are not forbidden by the MessagePack standard, many serializers impose this restriction. (It also means _any_ well-formed struct can be decoded into a mapinterface{}.) The only exception to this rule is that the decoders will allow you to read map keys encoded as bin types, since some legacy encodings permitted this. (However, those values will still be cast to Go strings, and they will be converted to str types when re-encoded. It is the responsibility of the user to ensure that map keys are UTF-8 safe in this case.) The same rules hold true for JSON translation. If the output compiles, then there's a pretty good chance things are fine. (Plus, we generate tests for you.) Please file an issue if you think the generator is writing broken code. Performance If you like benchmarks, see here , here , and here . CREDITS This repository is a fork of github.com/tinylib/msgp. Differences between this tool and tinylib/msgp: * Here we have regular expression matching for type names in directives. * Here we do not use package unsafe for conversions from byte slices to strings: byte is converted quite efficiently to string simply with the built-in string(). * This codebase is thoroughly refactored to be more Go-idiomatic and efficient. You're welcome to contribute! CHERRYWOODS/SWIFT-MSGPACK-SERIALIZATION HTTPS://GITHUB.COM/CHERRYWOODS/SWIFT-MSGPACK-SERIALIZATION SWIFT-MSGPACK-SERIALIZATION Another swift serialization library for msgpack with support for the Codable environement. This framework is designed to be (optionaly) compatibel to the msgpack-java library. See Compatibility.md for more detailed information about this. Please make sure, that the swift model classes and java model classes you use are compatibel to each other, e.g. enums are named equaly (including case). This framework also supplies the possibiliy to encode keys of other types than String (generalContainer function of MsgpackEncoder). swift-msgpack-serialization works with MessagePack.swift . This means, that you can encode your classes, structs and enums, that conform to Codable to MessagePackValue from MessagePack.swift, do something with this MessagePackValue and then encode it to msgpack using MessagePack.swift. You may of course also decode to MessagePackValue first and then decode a particular MessagePackValue to your own type. INSTALATION Currently, only carthage is supported. COMPATIBILTY RESTRICTIONS Make sure you never call .encode(to:) directly! Instead use the encode method on a keyed, unkeyed or singleValue container or cast the encoder passed to you to MsgpackEncoder and call encodeIntermediate(_) on this encoder. If you do not use this, this framework won't be able to do it's work properly. BASTIBE/MATLAB-MSGPACK HTTPS://GITHUB.COM/BASTIBE/MATLAB-MSGPACK A MESSAGEPACK IMPLEMENTATION FOR MATLAB The code is written in pure Matlab, and has no dependencies beyond Matlab itself. It does not work in Octave, since Octave does not support unicode2native or native2unicode. The files in this repository are taken from Transplant . BASIC USAGE: data = {'life, the universe, and everything', struct('the_answer', 42)}; bytes = dumpmsgpack(data) data = parsemsgpack(bytes) % returns: {'life, the universe, and everything', containers.Map('the_answer', 42)} CONVERTING MATLAB TO MSGPACK: MATLAB MSGPACK string string scalar number logical true/false vector array of numbers uint8 vector bin matrix array of array of numbers empty matrix nil cell array array cell matrix array of arrays struct map containers.Map map struct array array of maps handles raise error There is no way of encoding exts CONVERTING MSGPACK TO MATLAB MSGPACK MATLAB string string number scalar true/false logical nil empty matrix array cell array map containers.Map bin uint8 ext uint8 Note that since structs don't support arbitrary field names, they can't be used for representing maps. We use containers.Map instead. TESTS runtests() LICENSE MATLAB (R) is copyright of the Mathworks Copyright (c) 2014 Bastian Bechtold All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. DAVETHOMAS11/MOSHIPACK HTTPS://GITHUB.COM/DAVETHOMAS11/MOSHIPACK MOSHIPACK Gradle implementation com.daveanthonythomas.moshipack:moshipack:1.0.0-beta Optional Retrofit support: implementation com.daveanthonythomas.moshipack:moshipack-retrofit:1.0.0-beta ABOUT This is a Kotilin implementation of MessagePack serialization and deserialization built ontop of Moshi to take advantage of Moshi's type adapters and utilizes okio for reading and writing MessagePack bytes. The library is intended to be consumed in a Kotlin project, and is not intended for Java use. Inspired by Kaushik Gopal's tweet See Moshi for adapter usage and reference. Convert an object to MessagePack format data class MessagePackWebsitePlug(var compact: Boolean = true, var schema: Int = 0) val moshiPack = MoshiPack() val packed: BufferedSource = moshiPack.pack(MessagePackWebsitePlug()) println(packed.readByteString().hex()) This prints the MessagePack bytes as a hex string 82A7636F6D70616374C3A6736368656D6100 * 82 - Map with two entries * A7 - String of seven bytes * 63 6F 6D 70 61 63 74 - UTF8 String "compact" * C3 - Boolean value true * A6 - String of size bytes * 73 63 68 65 6D 61 - UTF8 String "schema" * 00 - Integer value 0 Convert binary MessagePack back to an Object val bytes = ByteString.decodeHex("82a7636f6d70616374c3a6736368656d6100").toByteArray() val moshiPack = MoshiPack() val plug: MessagePackWebsitePlug = moshiPack.unpack(bytes) Static API If you prefer to not instantiate a MoshiPack instance you can access the API in a static fashion as well. Note this will create a new Moshi instance every time you make an API call. You may want to use the API this way if you aren't providing MoshiPack by some form of dependency injection and you do not have any specific builder parameters for Moshi ------------------------- FORMAT SUPPORT See MessagePack format spec for further reference. FORMAT NAME FIRST BYTE (IN BINARY) FIRST BYTE (IN HEX) SUPPORTED positive fixint 0xxxxxxx 0x00 - 0x7f Yes fixmap 1000xxxx 0x80 - 0x8f Yes fixarray 1001xxxx 0x90 - 0x9f Yes fixstr 101xxxxx 0xa0 - 0xbf Yes nil 11000000 0xc0 Yes (never used) 11000001 0xc1 Yes false 11000010 0xc2 Yes true 11000011 0xc3 Yes bin 8 11000100 0xc4 No bin 16 11000101 0xc5 No bin 32 11000110 0xc6 No ext 8 11000111 0xc7 No ext 16 11001000 0xc8 No ext 32 11001001 0xc9 No float 32 11001010 0xca Yes float 64 11001011 0xcb Yes uint 8 11001100 0xcc Yes uint 16 11001101 0xcd Yes uint 32 11001110 0xce Yes uint 64 11001111 0xcf Yes int 8 11010000 0xd0 Yes int 16 11010001 0xd1 Yes int 32 11010010 0xd2 Yes int 64 11010011 0xd3 Yes fixext 1 11010100 0xd4 No fixext 2 11010101 0xd5 No fixext 4 11010110 0xd6 No fixext 8 11010111 0xd7 No fixext 16 11011000 0xd8 No str 8 11011001 0xd9 Yes str 16 11011010 0xda Yes str 32 11011011 0xdb Yes array 16 11011100 0xdc Yes array 32 11011101 0xdd Yes map 16 11011110 0xde Yes map 32 11011111 0xdf Yes negative fixint 111xxxxx 0xe0 - 0xff Yes ------------------------- API pack Serializes an object into MessagePack. RETURNS: okio.BufferedSource Instance version: MoshiPack().pack(anyObject) Static version: MoshiPack.pack(anyObject) packToByeArray If you prefer to get a ByteArray instead of a BufferedSource you can use this method. Instance version only MoshiPack().packToByteArray(anObject) Static can be done MoshiPack.pack(anObject).readByteArray() unpack Deserializes MessagePack bytes into an Object. RETURNS: T: Any Works with ByteArray and okio.BufferedSource Instance version: // T must be valid type so Moshi knows what to deserialize to val unpacked: T = MoshiPack().unpack(byteArray) Static version: val unpacked: T = MoshiPack.upack(byteArray) Instance version: val unpacked: T = MoshiPack().unpack(bufferedSource) Static version: val unpacked: T = MoshiPack.upack(bufferedSource) T can be an Object, a List, a Map, and can include generics. Unlike Moshi you do not need to specify a parameterized type to deserialize to a List with generics. MoshiPack can infer the paramterized type for you. The following examples are valid for MoshiPack: A typed List val listCars: List = MoshiPack.unpack(carMsgPkBytes) A List of Any val listCars: List = MoshiPack.unpack(carMsgPkBytes) An Object val car: Car = MoshiPack.unpack(carBytes) A Map of Any, Any val car: Map = MoshiPack.unpack(carBytes) msgpackToJson Convert directly from MessagePack bytes to JSON. Use this method for the most effecient implementation as no objects are instantiated in the process. This uses the FormatInterchange class to match implementations of JsonReader and a JsonWriter. If you wanted to say support XML as a direct conversion to and from, you could implement Moshi's JsonReader and JsonWriter classes and use the FormatInterchange class to convert directly to other formats. RETURNS String containing a JSON representation of the MessagePack data Instance versions: (takes ByteArray or BufferedSource) MoshiPack().msgpackToJson(byteArray) MoshiPack().msgpackToJson(bufferedSource) Static versions: (takes ByteArray or BufferedSource) MoshiPack.msgpackToJson(byteArray) MoshiPack.msgpackToJson(bufferedSource) jsonToMsgpack Convert directly from JSON to MessagePack bytes. Use this method for the most effecient implementation as no objects are instantiated in the process. RETURNS BufferedSource Instance versions: (takes String or BufferedSource) MoshiPack().jsonToMsgpack(jsonString) MoshiPack().jsonToMsgpack(bufferedSource) Static versions: (takes String or BufferedSource) MoshiPack.jsonToMsgpack(jsonString) MoshiPack.jsonToMsgpack(bufferedSource) MoshiPack - constructor + Moshi builder The MoshiPack constructor takes an optional Moshi.Builder.() -> Unit lambda which is applied to the builder that is used to instantiate the Moshi instance it uses. Example adding custom adapter: val moshiPack = MoshiPack({ add(customAdapter) }) Moshi is also a settable property which can be changed on a MoshiPack instance: val m = MoshiPack() m.moshi = Moshi.Builder().build() The static version of the API also can be passed a lambda to applied to the Moshi.Builder used to instantiate Moshi: MoshiPack.pack(someBytes) { add(customAdapter) } ------------------------- KOTILN SUPPORT Since this library is intended for Kotlin use, the moshi-kotlin artifact is included as a depedency. A KotlinJsonAdapterFactory is added by default to the instantiated Moshi that MoshiPack uses. This adapter allows for the use of Moshi's annotaions in Kotlin. To learn more about it see the Moshi documentation. If you'd like to use Moshi with out a KotlinJsonAdapterFactory supply a Moshi instance for MoshiPack: MoshiPack(moshi = Moshi.Builder().build) PROGUARD From Moshi's README.md; If you are using ProGuard you might need to add the following options: -dontwarn okio.** -dontwarn javax.annotation.** -keepclasseswithmembers class * { @com.squareup.moshi.* ; } -keep @com.squareup.moshi.JsonQualifier interface * -keepclassmembers class kotlin.Metadata { public ; } RETROFIT An example of using the retorfit adapter can be found here: https://github.com/davethomas11/MoshiPack_AndroidAppExample MRADZIWO/MSGPACK_LABVIEW HTTPS://GITHUB.COM/MRADZIWO/MSGPACK_LABVIEW MSGPACK_LAB-VIEW MsgPack_Lab-VIEW is a lightweight MessagePack serializer and deserializer library written in native LabVIEW 2016. One can decode a MsgPack message to LabVIEW variant containing correct type, or - when type is known at compile time - one can use dedicated decode functions for each supported type. PKCSECURITY/MSGPACK-CLJS HTTPS://GITHUB.COM/PKCSECURITY/MSGPACK-CLJS MSGPACK-CLJS A Clojurescript implementation of msgpack. This library's structure is _heavily_ adapted from the great work done on clojure-msgpack . Support for Clojure keyword, symbol, char, ratio, and set are all included by default as Extension types for convenience and to be compatible with clojure-msgpack. USAGE pack takes an arbitrary object and converts it into a Uint8Array. unpack takes a Uint8Array and converts it back into an arbitrary object. IMPROVEMENTS This library could stand to be refactored. We just use this internally at PKC , so, if there is demand (or interest) in making this more feature-rich, we are open to making improvements. Just be vocal, otherwise we'll keep it simple. COPYRIGHT Copyright 2018 PKC Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at: http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. DANKOGAI/SWIFT-SION HTTPS://GITHUB.COM/DANKOGAI/SWIFT-SION SION SUPPORTS MESSAGEPACK NATIVELY SYNOPSIS import SION import Foundation let data = Data() let sion = SION(msgPack: data) // deselialize let msgPack = sion.msgPack // serialize data == msgPack // true for details of the module, visit: * https://github.com/dankogai/swift-sion for details of the SION serialization format, visit: * https://dankogai.github.io/SION YGOE/MSGPACK.JS HTTPS://GITHUB.COM/YGOE/MSGPACK.JS MSGPACK.JS This is a MessagePack serializer and deserializer written in JavaScript for web browsers (including IE 11) and Node.js. It is compact but still fully-featured. This library supports the complete MessagePack specification released on 2017-08-09, including date/time values. No other extension types are implemented in this library, it’s only the standard types which is perfectly fine for interoperability with MessagePack codecs in other programming languages. I’m using the MessagePack-CSharp library on the server side in my .NET applications. MESSAGEPACK MessagePack is an efficient binary serialisation format. It lets you exchange data among multiple languages like JSON. But it’s faster and smaller. Small integers are encoded into a single byte, and typical short strings require only one extra byte in addition to the strings themselves. SIZE This library is very lightweight. The source code has around 510 LINES (incl. browser/Node detection), the minified file has 6.6 kB and can be GZip-compressed to 2.4 KB. PERFORMANCE The file msgpack-tests.html contains some tests and a benchmark function that compares this library with msgpack-lite . Here are the results, in milliseconds (lower is better). All tests done on an Intel Core i7-3770 and Windows 10. FUNCTION CHROME 72 FIREFOX 65 EDGE 16 IE 11 MSGPACK.JS SERIALIZE 702 ms +6% 1232 ms −42% 2483 ms +41% 2493 ms −3% msgpack-lite encode 663 ms 2124 ms 1762 ms 2578 ms MSGPACK.JS DESERIALIZE 652 ms +13% 869 ms +5% 821 ms −48% 651 ms −68% msgpack-lite decode 577 ms 827 ms 1587 ms 2021 ms The numbers show that this library is comparable with msgpack-lite. In Chrome it’s only 10% slower. But serializing in Firefox and deserializing in Microsoft browsers is twice as fast. USAGE Browser In browsers, a global msgpack object is created that contains the functions serialize and deserialize. The first can be called with any data and returns the serialized bytes. The second works in reverse, taking the serialized bytes and returning the runtime value. Include the JavaScript file into your HTML document like this: You can use the library functions after loading the script. If there should be a naming conflict with another library you want to load, you can change the global object name from msgpack to something else by setting msgpackJsName before loading the script file: Node.js In Node.js, these functions are exported in the object you get from the require function. var msgpack = require('@ygoe/msgpack'); Example Here’s a simple example: // Define some data var sourceData = { number: 123, number2: -0.129, text: "Abc with Üñıçôðé and ユニコード", flag: true, list: , obj: { a: 1, b: "2", c: false, d: { a: 0, b: -1 } }, time: Date.now() }; // Serialize to byte array var bytes = msgpack.serialize(sourceData); // Deserialize again var deserializedData = msgpack.deserialize(bytes); Compatibility You can also use the functions encode and decode which are aliases to serialize and deserialize. This makes it easier to replace other libraries that use these function names with msgpack.js. New projects should use the preferred (and more precisely named) serialize and deserialize functions though. LICENSE MIT license SHAMATON/MSGPACK HTTPS://GITHUB.COM/SHAMATON/MSGPACK MESSAGEPACK FOR GOLANG * Supported types : primitive / array / slice / struct / map / interface{} and time.Time * Renames fields via msgpack:"field_name" * Ignores fields via msgpack:"ignore" * Supports extend encoder / decoder * Can also Encoding / Decoding struct as array INSTALLATION go get -u github.com/shamaton/msgpack QUICK START package main; import ( "github.com/shamaton/msgpack" ) func main() { type Struct struct { String string } v := Struct{String: "msgpack"} d, err := msgpack.Encode(v) if err != nil { panic(err) } r := Struct{} err = msgpack.Decode(d, &r) if err != nil { panic(err) } } BENCHMARK This result made from shamaton/msgpack_bench Encode BenchmarkCompareEncodeShamaton-8 1000000 1319 ns/op 320 B/op 3 allocs/op BenchmarkCompareEncodeShamatonArray-8 1000000 1172 ns/op 256 B/op 3 allocs/op BenchmarkCompareEncodeVmihailenco-8 300000 4626 ns/op 1000 B/op 14 allocs/op BenchmarkCompareEncodeVmihailencoArray-8 500000 3918 ns/op 680 B/op 13 allocs/op BenchmarkCompareEncodeUgorji-8 1000000 1985 ns/op 986 B/op 11 allocs/op BenchmarkCompareEncodeJson-8 500000 3649 ns/op 1224 B/op 16 allocs/op BenchmarkCompareEncodeGob-8 100000 12324 ns/op 2824 B/op 50 allocs/op Decode BenchmarkCompareDecodeShamaton-8 1000000 1494 ns/op 512 B/op 6 allocs/op BenchmarkCompareDecodeShamatonArray-8 1000000 1031 ns/op 512 B/op 6 allocs/op BenchmarkCompareDecodeVmihailenco-8 200000 5660 ns/op 1056 B/op 33 allocs/op BenchmarkCompareDecodeVmihailencoArray-8 300000 4779 ns/op 992 B/op 22 allocs/op BenchmarkCompareDecodeUgorji-8 500000 2774 ns/op 844 B/op 12 allocs/op BenchmarkCompareDecodeJson-8 200000 9721 ns/op 1216 B/op 43 allocs/op BenchmarkCompareDecodeGob-8 50000 37553 ns/op 10172 B/op 275 allocs/op LICENSE This library is under the MIT License. KIESELSTEINI/MSGPACK HTTPS://GITHUB.COM/KIESELSTEINI/MSGPACK MESSAGEPACK FOR LUA 5.3 OVERVIEW This is a pure Lua implementation for encoding/decoding MessagePack (https://msgpack.org). Please report any bugs you encounter! Features: * written in pure Lua 5.3 (using string.pack() / string.unpack()) * can distinguish between integer / float numbers * can distinguish between UTF-8 strings and binary data * public domain license (http://unlicense.org ) * pretty fast decoding What's missing: * extendend types fixent Example code: local msgpack = require('msgpack') local value = msgpack.decode(binary_msgpack_data) -- decode to Lua value local binary_data = msgpack.encode(lua_value) -- encode Lua value to MessagePack API msgpack.encode_one(value) Encodes the given Lua value to a binary MessagePack representation. It will return the binary string on succes or nil plus an error message if it fails. The encoder will encode Lua strings as MessagePack strings when they are properly UTF-8 encoded otherwise they will become MessagePack binary objects. There is also a check if a Lua number can be lossless encoded as a 32-bit float. > NOTE: Empty Lua tables will be encoded as empty arrays! msgpack.encode(...) Encodes all given values to a binary MessagePack representation. It will return the binary string or nil plus an error message if it fails. local binary = msgpack.encode('Hello', 1024, true, { 2, 3, 4 }) msgpack.decode_one(binary_data) Decode the given MessagePack binary string to a corresponding Lua value. It will return the decoded Lua value and the position for next byte in stream or nil plus an error message if decoding went wrong. You can use the returned position to decode multiple MessagePack values in a stream. The optional position argument is used to start the decoding at a specific position inside the the binary_data string. > NOTE: Extended types are not supported. Decoding will fail! > NOTE: Binary data will be decoded as Lua strings > NOTE: Arrays will be decoded as Lua tables starting with index 1 > (like Lua uses tables as arrays) > NOTE: Values which are nil will cause the key, value pair to > disappear in a Lua table (that's how it works in Lua) msgpack.decode(binary_data) Decode the given MessagePack binary string to one or more Lua values. It will return all decoded Lua values or nil plus an error message if decoding failed. local a, b, c = msgpack.decode(binary) LICENSE This is free and unencumbered software released into the public domain. Anyone is free to copy, modify, publish, use, compile, sell, or distribute this software, either in source code form or as a compiled binary, for any purpose, commercial or non-commercial, and by any means. In jurisdictions that recognize copyright laws, the author or authors of this software dedicate any and all copyright interest in the software to the public domain. We make this dedication for the benefit of the public at large and to the detriment of our heirs and successors. We intend this dedication to be an overt act of relinquishment in perpetuity of all present and future rights to this software under copyright law. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. For more information, please refer to INSTALLATION Use the included makefile to build the library, documentation and run tests. Here is a quick run-down, please refer to the INSTALL file for full details. # Run tests, install to /usr/local, build PDF and HTML documentation make check && make lib doc PREFIX=/usr/local && make pdf html USING MESSAGEPACK First import the module, then you can pack an object to a binary input port or unpack one from a binary output port. (use-modules ((msgpack) #:select (pack unpack pack-to unpack-from))) ;; Some object to pack (define hodgepodge (vector 1 2 '#(3 #t) "foo")) (pack hodgepodge) ;; Returns #vu8(#x94 #x01 #x02 #x92 #x03 #xC3 #xA3 #x66 #x6F #x6F) (unpack #vu8(#x94 #x01 #x02 #x92 #x03 #xC3 #xA3 #x66 #x6F #x6F)) ;; Returns '#(1 2 #(3 #t) "foo") ;; We can also pack and unpack directly to and from a port (pack-to (current-output-port) hodgepodge) (unpack-from (current-input-port)) In the above code we have two variants of pack and unpack: a pure one and an impure one with a to- or from suffix. The pure variant performs the packing or unpacking without any side effects by returning the packed bytes or the unpacked object. The impure variant on the other hand writes to or reads from a port as a side effect, but does not return anything. STATUS OF THE PROJECT The project is still fresh, but it is complete and stable as far as I am concerned. I will wait a little while for it to set before committing to a proper release though. User experience reports are always welcome. LICENSE Released under the GPLv3+ license, see the COPYING file for details. SELF-PROMOTION If you like this library please consider financially supporting its development, every small amount helps. Feel free to explore my other software projects as well. * http://hiphish.github.io/ * https://liberapay.com/HiPhish/ DAVALAPAR/WHAT-THE-PACK HTTP://NPMJS.COM/PACKAGE/WHAT-THE-PACK WHAT-THE-PACK Ultra-fast MessagePack for NodeJS & Browsers. IMPLEMENTATION NOTES * this implementation uses pre-allocated buffers and buffer.copy() for encoding, instead of regular arrays * uses a buffer polyfill if used in browser environments * has dictionary support, to further reduce payload size BACKWARD COMPATIBILITY NOTES WITH OTHER LIBRARIES * used extensions * fixext 1, type 0, data 0 = undefined * fixext 1, type 0, data 1 = NaN * fixext 1, type 0, data 2 = +Infinity * fixext 1, type 0, data 3 = -Infinity * Buffers, ArrayBuffers and TypedArrays * Buffers : encoded as Buffers, decoded as Buffers * ArrayBuffers : encoded as Buffers, decoded as Buffers const decoded = decode(encoded); const your_arraybuffer = decoded.buffer; * TypedArrays : encoded as Buffers, decoded as Buffers const decoded = decode(encoded); const your_typedarray = new Uint8Array(decoded.buffer); USAGE yarn add what-the-pack const MessagePack = require('what-the-pack'); const { encode, decode } = MessagePack.initialize(2**22); // 4MB const data = { name: 'Lunox', age: 20 }; const encoded = encode(data); const decoded = decode(encoded); console.log({ encoded, decoded }); RESULT { encoded: , decoded: { name: 'Lunox', age: 20 } } PRE-ALLOCATING A LARGER BUFFER const MessagePack = require('what-the-pack'); const { encode, decode } = MessagePack.initialize(2**30); // 1GB const data = { // large data goes here }; 2^7 = 128 B 2^8 = 256 B 2^9 = 512 B 2^10 = 1.02 kB 2^11 = 2.05 kB 2^12 = 4.1 kB 2^13 = 8.19 kB 2^14 = 16.4 kB 2^15 = 32.8 kB 2^16 = 65.5 kB 2^17 = 131 kB 2^18 = 262 kB 2^19 = 524 kB 2^20 = 1.05 MB 2^21 = 2.1 MB 2^22 = 4.19 MB 2^23 = 8.39 MB 2^24 = 16.8 MB 2^25 = 33.6 MB 2^26 = 67.1 MB 2^27 = 134 MB 2^28 = 268 MB 2^29 = 537 MB 2^30 = 1.07 GB USING DICTIONARIES (ADDED IN 1.1.3) * this feature isn't in MessagePack spec but added as a convenience feature in 1.1.3 * dictionaries allow us to decrease our buffer output size by recognizing strings used as object keys and replacing them with shorter-byte integer values during the encoding process * these shorter-byte placeholder values are then restored to their respective strings during the decoding process * the trade-off in using dictionaries is an insignificantly slower encoding and decoding time in exchange of a significantly smaller buffer output, which results into a lower network bandwidth and storage consumption in the long run * the best part: the byte placeholders starts from -32 then increments upwards, values -32 to 127 are encoded in single byte, which means your first (32 + 128) = 160 keys will be encoded as a single byte instead of encoding the whole string const MessagePack = require('what-the-pack'); const { encode, decode, register } = MessagePack.initialize(2**22); // 4MB let encoded, decoded, data; data = { name: 'Lunox', age: 20 }; encoded = encode(data); decoded = decode(encoded); console.log({ encoded, decoded }); /** - encoded: (17) - decoded: { name: 'Lunox', age: 20 } **/ register('name', 'age'); encoded = encode(data); decoded = decode(encoded); console.log({ encoded, decoded }); /** - encoded: (10) - decoded: { name: 'Lunox', age: 20 } **/ MINIFIED BUILD FOR BROWSERS server const WebSocket = require('ws'); const MessagePack = require('what-the-pack'); const { encode, decode } = MessagePack.initialize(2**22); // 4MB const wss = new WebSocket.Server( /- options go here */ ); wss.on('connection', (client, req) => { console.log('A client has connected.'); console.log('IP address:', req.connection.remoteAddress); client.send( encode({ message: 'something' }) ); }); client * On browsers, Buffer object is exposed as MessagePack.Buffer * On browsers, call MessagePack.Buffer.from(x) on received ArrayBuffers // Create WebSocket connection. const socket = new WebSocket('ws://localhost:8080'); const { encode, decode, Buffer } = MessagePack.initialize(2**22); // 4MB // Connection opened socket.addEventListener('open', (event) => { socket.binaryType = 'arraybuffer'; // important console.log('Connected to server.'); }); // Listen for messages socket.addEventListener('message', (event) => { const data = MessagePack.decode( Buffer.from(event.data) ); console.log(data); // logs: { message: 'something' } }); BENCHMARKS yarn run benchmark $ yarn run benchmark JSON stringify tiny x 1,477,866 ops/sec ±0.58% (93 runs sampled) JSON stringify small x 232,645 ops/sec ±0.25% (91 runs sampled) JSON stringify medium x 117,357 ops/sec ±0.31% (93 runs sampled) JSON stringify large x 24.01 ops/sec ±0.37% (43 runs sampled) JSON parse tiny x 1,301,925 ops/sec ±3.18% (82 runs sampled) JSON parse small x 264,410 ops/sec ±0.57% (90 runs sampled) JSON parse medium x 133,865 ops/sec ±0.52% (87 runs sampled) JSON parse large x 31.52 ops/sec ±0.34% (53 runs sampled) what-the-pack encode tiny x 1,175,981 ops/sec ±0.39% (92 runs sampled) what-the-pack encode small x 365,533 ops/sec ±0.85% (90 runs sampled) what-the-pack encode medium x 173,746 ops/sec ±0.41% (91 runs sampled) what-the-pack encode large x 218 ops/sec ±0.85% (82 runs sampled) what-the-pack decode tiny x 1,130,260 ops/sec ±0.30% (91 runs sampled) what-the-pack decode small x 254,931 ops/sec ±0.79% (94 runs sampled) what-the-pack decode medium x 146,809 ops/sec ±0.79% (92 runs sampled) what-the-pack decode large x 211 ops/sec ±0.37% (87 runs sampled) notepack.encode tiny x 1,291,361 ops/sec ±0.22% (95 runs sampled) notepack encode small x 325,882 ops/sec ±1.20% (95 runs sampled) notepack encode medium x 133,398 ops/sec ±0.20% (94 runs sampled) notepack encode large x 231 ops/sec ±1.65% (81 runs sampled) notepack decode tiny x 1,097,597 ops/sec ±0.67% (93 runs sampled) notepack decode small x 231,895 ops/sec ±0.69% (96 runs sampled) notepack decode medium x 137,385 ops/sec ±2.45% (86 runs sampled) notepack decode large x 210 ops/sec ±0.85% (86 runs sampled) TESTS yarn run test $ yarn run test PASS ./test.js √ fixstr (6ms) √ str 8 (2ms) √ str 16 (1ms) √ str 32 (1ms) √ zero √ positive fixint (1ms) √ negative fixint (1ms) √ uint 8 (1ms) √ uint 16 (1ms) √ uint 32 √ uint 64 (1ms) √ int 8 (1ms) √ int 16 √ int 32 (2ms) √ int 64 √ float 32 (2ms) √ float 64 (1ms) √ true, false, undefined, NaN, +Infinity, -Infinity (2ms) √ flat & nested empty arrays (1ms) √ flat arrays (456ms) √ nested arrays (5ms) √ highly nested arrays (2ms) √ buffers, bin8 (2ms) √ buffers, bin16 (96ms) √ buffers, bin32 (473ms) √ arraybuffers as buffer (54ms) √ typedarrays as buffer (7ms) √ tiny object (1ms) √ small object √ medium object (1ms) √ large object (1736ms) console.log index.js:49 MessagePack: Setting buffer limit to 1.07 GB Test Suites: 1 passed, 1 total Tests: 31 passed, 31 total Snapshots: 0 total Time: 5.477s Ran all test suites. Done in 6.59s. CHANGELOG * 1.x * basic support * dictionary support * 2.0.0 * rewrite to use raw functions instead of classes * update dev-deps * jest test-cov @ 86.06% * statements 389/452 * branches 137/169 * functions 11/12 * lines 374/428 * 2.0.x * fix tempBufferLength check * rebuild for browser * fix leak on buffer decode REFERENCES * buffer re-alloc idea * https://github.com/darrachequesne/notepack/issues/12#issuecomment-320872590 (Manuel Astudillo) * notepack.io * https://github.com/darrachequesne/notepack (MIT, Damien Arrachequesne) * notepack * https://github.com/hypergeometric/notepack (MIT, Ben Shepheard) * buffer: * https://github.com/feross/buffer (MIT, Feross Aboukhadijeh) * pretty-bytes * https://github.com/sindresorhus/pretty-bytes (MIT, Sindre Sorhus) MIT | @davalapar KNOPP/MSGPACK_DART HTTPS://GITHUB.COM/KNOPP/MSGPACK_DART MSGPACK_DART MessagePack implementation for dart. Clean, simple, fast and with sane API and implementation. UGORJI/GO-CODEC HTTPS://GITHUB.COM/UGORJI/GO-CODEC GO-CODEC idiomatic codec and rpc lib for msgpack, cbor, json, etc. mirrors github.com/ugorji/go/codec . msgpack.org HIROTAKAN/MESSAGEPACKER HTTPS://GITHUB.COM/HIROTAKAN/MESSAGEPACKER MESSAGEPACKER MESSAGEPACKER is a MessagePack encoder & decoder for Swift and supports Codable . * Message Pack specification: https://github.com/msgpack/msgpack/blob/master/spec.md USAGE import MessagePacker struct Coordinate: Codable { var latitude: Double var longitude: Double } struct Landmark: Codable { var name: String var foundingYear: Int var location: Coordinate } let input = Landmark( name: "Mojave Desert", foundingYear: 0, location: Coordinate( latitude: 35.0110079, longitude: -115.4821313 ) ) let data = try! MessagePackEncoder().encode(input) let landmark = try! MessagePackDecoder().decode(Landmark.self, from: data) print((data)) print(landmark) // // Landmark( // name: "Mojave Desert", // foundingYear: 0, // location: MessagePackerTests.Coordinate( // latitude: 35.0110079, // longitude: -115.4821313 // ) // ) INSTALLATION Carthage Add the following to your Cartfile: github "hirotakan/MessagePacker" CocoaPods Add the following to your Podfile: pod 'MessagePacker' SwiftPM Add MessagePacker as a dependency: import PackageDescription let package = Package( name: "YourApp", dependencies: ) REQUIREMENTS * Swift 4.1 or later * iOS 8.0 or later * macOS 10.10 or later * tvOS 9.0 or later * watchOS 2.0 or later LICENSE ImageTransition is released under the MIT license. See LICENSE for details. RELATED PROJECTS ZERORPC BY DOTCLOUD zerorpc is a flexible RPC implementation based on zeromq and messagepack. Service APIs exposed with zerorpc are called "zeroservices". PFICOMMON BY PREFERRED INFRASTRUCTURE, INC. General purpose C++ library for Preferred Infrastructure, Inc. It includes MessagePack-RPC implementation for C++ JSON 27 BYTES {"compact":true,"schema":0} MESSAGEPACK (HEX) 18 BYTES 67 % 82 a7 63 6f 6d 70 61 63 74 c3 a6 73 63 68 65 6d 61 00 MessagePack ™ Copyright © 2008-2019 Sadayuki Furuhashi 
