This blurb is about serialization for QDL.

3/1/2022

There is still Java serialization, but that is fragile and, except in a few specific use cases,
should be avoided. Mostly this is because any changes to the state (frequent between
versions) render the saved workspaces unreadable.

XML serialization is what we have settled on.

Version 1 XML:
--------------
This saved lots of things directly, so there were copies of functions etc.
repeatedly everywhere. For smaller workspaces this was not an issue, but it became
clear that complex workspaces were often enormous for no good reason (several hundred
kilobytes compressed).

A major revision to the system in Jan/Feb 2022 was to make modules and variables use
the new XStack machinery. The switch was needed because a consistent API for managing
stacks of things was needed. Before variables and functions had their own stack code
that was managed in parallel. This mostly happened by a process of evolution, but it was
clear that this was going to be a daunting and confusing maintenance task if it continued.
On the whole, designing it all cleanly and replacing everything was the order of the day.

This also meant that rather than have global module templates (and instances), these
could/should be in stacks. Since instances have state, suddenly old XML serialization
exploded and at times would fail to work completely because very complex workspaces
might contain cyclic references to state objects.

Version 2 of XML:
-----------------
This has lists of state objects and templates
which are outside of the serialization proper, and references to these master lists
are then done. This prevents recursion and cycles. There is more overhead for setting this
up. This also means that for very small
workspaces, the size is larger than version 1, but for extremely large workspaces it
ends up being much smaller.

Version 2 Design:
-----------------
* Do not serialize data structures (like lists, stems). That gets large and complex
  and easily breaks.
* Serialize QDL source code (modules, functions, variables)
* Use references to shared state. There is one serialization of any state object.
* Use XML to capture the structure around the QDL, such as what is in stacks, current
  settings for variables (e.g. numeric digits)
* Interpret QDL source and mine the references from a local scratch state instance.
  >> This is critical since QDL can be very, very complex indeed and the problem with
     Version 1 boiled down to attempting to capture this. The serialization was in
     effect rewriting the parser from scratch. No. We wrote a parser for a good reason,
     let it do the work.
* Lists of things (such as the users preferences, lib paths etc.) are turned into JSON
  objects and serialized as base 64 encoded strings. This is really what JSON was
  created for anyway. It gives a ready-made API for managing types (booleans, ints etc)
  and hides some of the complexity so I don't have to re-invent the wheel in XML
  to stash some flags.
* Any string gets base 64 encoded, since we have no control over content and a user
  could set a variable like a:=']]>' (the XML CDATA close tag) and kill the whole
  system. XML libraries will let you write illegal XML happily but it cannot be
  read then. Base64 encoding solves that completely.
* Practical downside is that for huge workspaces, all of the base64 encoding
  (specifically of huge stems) can
  be s-l-o-w. If it turns out to be an actual issue, this may need to be revisited.

Adding attributes and elements.
-------------------------------
There are four methods called read/write attributes/elements. These are invoked
in XStack, XTable etc. as needed. Generally for reading attributes and elements,
just check the tag is correct.

Reading and writing Elements:
-----------------------------


    public void writeExtraXMLElements(XMLStreamWriter xsr) throws XMLStreamException {
        xsr.writeStartElement(STATE_CONSTANTS_TAG);
        JSONObject json = new JSONObject();
        json.put(STATE_ASSERTIONS_ENABLED_TAG, isAssertionsOn());
        json.put(STATE_PID_TAG, getPID());
        json.put(STATE_RESTRICTED_IO_TAG, isRestrictedIO());
        json.put(STATE_SERVER_MODE_TAG, isServerMode());
        // Saving the numeric digits this way is unsatisfactory since it is also done in all
        // of the sub-states, but there is no easy way to set this once and get it right
        // The top-level state is the last read, so it wil always end up getting set
        // correctly at the end. Best we can do without a ton of machinery...
        json.put(STATE_NUMERIC_DIGITS_TAG, OpEvaluator.getNumericDigits());
        xsr.writeCData(Base64.encodeBase64URLSafeString(json.toString().getBytes(StandardCharsets.UTF_8)));
        xsr.writeEndElement();
    }

Note that write is invoked once per serialization, so no need to test the tag.
For reading the elements, check the tag:

    public void readExtraXMLElements(XMLEvent xe, XMLEventReader xer) throws XMLStreamException {
        if(xe.asStartElement().getName().getLocalPart().equals(STATE_CONSTANTS_TAG)) {
            // only process the tag if it is the right one
            String text = XMLUtilsV2.getText(xer, STATE_CONSTANTS_TAG);
            text = new String(Base64.decodeBase64(text));
            JSONObject json = JSONObject.fromObject(text);
            setAssertionsOn(json.getBoolean(STATE_ASSERTIONS_ENABLED_TAG));
            setPID(json.getInt(STATE_PID_TAG));
            setServerMode(json.getBoolean(STATE_SERVER_MODE_TAG));
            setRestrictedIO(json.getBoolean(STATE_RESTRICTED_IO_TAG));
            OpEvaluator.setNumericDigits(json.getInt(STATE_NUMERIC_DIGITS_TAG));
        }
    }

The contract is that when an unknown tag is encountered, this method gets invoked,
so it will get invoked many times. Therefore this needs to check that the tag found
is the right one. You can do that with a conditional or a switch statement.

Reading and writing attributes (to the main tag):
------------------------------------------------
In those classes with read/writeExtraAttributes, the contract is that
write is invoked exactly once right after the main tag (e.g. <module>)
so do not move the XML cursor, just write attributes.
Read is invoked exactly once as well on the main tag, so
do not move the XML cursor, just read.

E.g. JavaModule
    public void writeExtraXMLAttributes(XMLStreamWriter xsw) throws XMLStreamException {
        super.writeExtraXMLAttributes(xsw);
        xsw.writeAttribute(XMLConstants.MODULE_TYPE_TAG, XMLConstants.MODULE_TYPE_JAVA_TAG);
        xsw.writeAttribute(XMLConstants.MODULE_CLASS_NAME_TAG, getClassname());
    }

However, reading these attributes is done in the main loop for Module (ok, special case)
since these are handling templates that are Java or native QDL at that point.