:py:mod:`gnome.persist` ======================= .. py:module:: gnome.persist .. autoapi-nested-parse:: Default behavior: Apply colander monkey patch by default Put all the common Schema nodes in one namespace Submodules ---------- .. toctree:: :titlesonly: :maxdepth: 1 base_schema/index.rst extend_colander/index.rst monkey_patch_colander/index.rst save_load/index.rst schema_decorator/index.rst validators/index.rst Package Contents ---------------- Classes ~~~~~~~ .. autoapisummary:: gnome.persist.Savable gnome.persist.References gnome.persist.TimeDelta gnome.persist.LocalDateTime gnome.persist.ObjType gnome.persist.LongLatBounds gnome.persist.WorldPoint gnome.persist.ImageSize Functions ~~~~~~~~~ .. autoapisummary:: gnome.persist.load gnome.persist.is_savezip_valid gnome.persist.convertible_to_seconds gnome.persist.ascending_datetime gnome.persist.no_duplicate_datetime gnome.persist.positive gnome.persist.now .. py:class:: Savable Bases: :py:obj:`object` Create a mixin for save/load options for saving and loading serialized gnome objects. Mix this in with the Serializable class so all gnome objects can save/load themselves .. py:attribute:: _allowzip64 :value: False .. py:method:: _ref_in_saveloc(saveloc, ref) returns true if reference is found in saveloc, false otherwise .. py:method:: _update_and_save_refs(json_, saveloc, references) for attributes that are stored as references - ensure the references are moved to saveloc, and update its value in the json_ .. py:method:: _json_to_saveloc(json_, saveloc, references=None, name=None) save json_ to saveloc -. first save and update references if any attributes are stored as references -. add self to references -. move any datafiles to saveloc that need to be persisted -. finally, write json_ directly to zip if saveloc is a zip or dump to *.json of saveloc is a directory :param json_: json data after serialization. Default is the output of self.serialize('save') :param saveloc: Either zipfile to which object's json can be appended. Or a valid path where object's *.json can be dumped to a file :param references: References object to keep track of objects that have been constructed. If a referenced object has been saved, no need to add it again to saveloc - don't want multiple copies :param name: json is stored in 'name.json' in zip archive/specified directory. Default is self.__class__.__name__. If references object contains self.__class__.__name__, then let .. py:method:: _write_to_file(saveloc, f_name, json_) .. py:method:: _write_to_zip(saveloc, f_name, s_data) general function for writing string data directly to zipfile. f_name is the archive name and s_data is the corresponding string, added to zipfile .. py:method:: _move_data_file(saveloc, json_) - Look at _state attribute of object. - Find all fields with 'isdatafile' attribute as True. - If there is a key in json_ corresponding with 'name' of the fields with True 'isdatafile' attribute - then - move that datafile and - update the key in the json_ to point to new location .. py:method:: _load_refs(json_data, saveloc, references) :classmethod: loads up references. First looks for object in references, if not found then it creates object from json and adds a reference to it. If found in references, it just uses/returns the object .. py:method:: _update_datafile_path(json_data, saveloc) :classmethod: update path to attributes that use a datafile if saveloc is a zipfile, then extract the datafile to same location as zipfile and upate path in json_data. .. py:method:: loads(json_data, saveloc=None, references=None) :classmethod: loads object from json_data - load json for references from files - update paths of datafiles if needed - deserialize json_data - and create object with new_from_dict() json_data: dict containing json data. It has been parsed through the json.loads() command. The json will be valided here when it gets deserialized. Its references and datafile paths will be recreated here prior to calling new_from_dict() Optional parameter :param saveloc: location of data files or .json files for objects stored as references. If object requires no datafiles and does not need to read references from a .json file in saveloc, then this can be None. :param references: references object - if this is called by the Model, it will pass a references object. It is not required. :returns: object constructed from json_data. .. py:method:: _save_collection(saveloc, coll_, refs, coll_json) Reference objects inside OrderedCollections or list. Since the OC itself isn't a reference but the objects in the list are a reference, do something a little differently here :param OrderedCollection coll_: ordered collection/list to be saved .. py:method:: _load_collection(saveloc, l_coll_dict, refs) :classmethod: doesn't need to be classmethod of the Model, but its only used by Model at present .. py:class:: References Bases: :py:obj:`object` PyGnome objects like the PointWindMover contain other objects, eg Wind object. When persisting a Model, the Wind object is not created by the PointWindMover, it is merely referenced by the PointWindMover. When persisting a Model, the referenced objects are saved in their own file and a reference is stored for it. This class manages these references. .. py:property:: files .. py:method:: __contains__(obj) .. py:method:: get_reference(obj) return key if obj already exists in references list else return None .. py:method:: _add_reference_with_name(obj, name) add an object reference specified by 'name' .. py:method:: reference(obj, name=None) Get a unique reference to the object. By default this string is the filename in which the json for the object is stored If a reference to obj already exists, then it is returned :param obj: object for which a reference must be added :param name=None: add an object reference specified by 'name' for filename .. py:method:: retrieve(ref) retrieve the object associated with the reference .. py:function:: load(saveloc, fname='Model.json', references=None) read json from file and load the appropriate object This is a general purpose load method that looks at the json['obj_type'] and invokes json['obj_type'].loads(json) method :param saveloc: path to zipfile that contains data files and json files. It could also be a directory containing files - keep original support for location files. :param fname: .json file to load. Default is 'Model.json'. zipfile/dir must contain 'fname' :param references=None: References object that keeps track of objects in a dict as they are constructed, using the filename as the key :returns: object constructed from the json .. note:: Function first assumes saveloc is a directory and looks for saveloc/fname. If this fails, it checks if saveloc is a zipfile. If this fails, it checks if saveloc is a file and loads this assuming its json for a gnome object. If none of these work, it just returns None. .. py:function:: is_savezip_valid(savezip) some basic checks on validity of zipfile. Primarily for checking save zipfiles loaded from the Web. Following are the types of errors it checks: :returns: True if zip is valid, False otherwise 1. Failed to open zipfile 2. CRC failed for a file in the archive - rejecting zip 3. Found a *.json with size > _max_json_filesize - rejecting 4. Reject - found a file with: uncompressed_size/compressed_size > _max_compress_ratio. 5. Found a file in archive that has path outside of saveloc - rejecting rejects zipfile if it contains an archive with '..' .. note:: can change _max_json_filesize, _max_compress_ratio if required. .. py:class:: TimeDelta Bases: :py:obj:`colander.Float` Add a type to serialize/deserialize timedelta objects .. py:method:: serialize(node, appstruct) .. py:method:: deserialize(*args, **kwargs) .. py:class:: LocalDateTime(*args, **kwargs) Bases: :py:obj:`colander.DateTime` A type representing a Python ``datetime.datetime`` object. This type serializes python ``datetime.datetime`` objects to a `ISO8601 `_ string format. The format includes the date, the time, and the timezone of the datetime. The constructor accepts an argument named ``default_tzinfo`` which should be a Python ``tzinfo`` object. If ``default_tzinfo`` is not specified the default tzinfo will be equivalent to UTC (Zulu time). The ``default_tzinfo`` tzinfo object is used to convert 'naive' datetimes to a timezone-aware representation during serialization. If ``default_tzinfo`` is explicitly set to ``None`` then no default tzinfo will be applied to naive datetimes. You can adjust the error message reported by this class by changing its ``err_template`` attribute in a subclass on an instance of this class. By default, the ``err_template`` attribute is the string ``Invalid date``. This string is used as the interpolation subject of a dictionary composed of ``val`` and ``err``. ``val`` and ``err`` are the unvalidatable value and the exception caused trying to convert the value, respectively. These may be used in an overridden err_template as ``${val}`` and ``${err}`` respectively as necessary, e.g. ``_('${val} cannot be parsed as an iso8601 date: ${err}')``. For convenience, this type is also willing to coerce ``datetime.date`` objects to a DateTime ISO string representation during serialization. It does so by using midnight of the day as the time, and uses the ``default_tzinfo`` to give the serialization a timezone. Likewise, for convenience, during deserialization, this type will convert ``YYYY-MM-DD`` ISO8601 values to a datetime object. It does so by using midnight of the day as the time, and uses the ``default_tzinfo`` to give the serialization a timezone. If the :attr:`colander.null` value is passed to the serialize method of this class, the :attr:`colander.null` value will be returned. The subnodes of the :class:`colander.SchemaNode` that wraps this type are ignored. .. py:method:: strip_timezone(_datetime) .. py:method:: serialize(node, appstruct) Serialize a DateTime object returns an iso formatted string .. py:method:: deserialize(node, cstruct) .. py:function:: convertible_to_seconds(node, value) validate only datetime objects .. py:function:: ascending_datetime(node, values) Check the datetime values in numpy structured array (like datetime_value_2d) are in ascending order .. py:function:: no_duplicate_datetime(node, values) Check for duplicate datetime values in numpy structured array like datetime_value_2d Reject ``values`` if it contains duplicates. .. py:function:: positive(node, value) .. py:class:: ObjType(unknown='ignore') Bases: :py:obj:`colander.SchemaType` Base class for all schema types .. py:attribute:: unknown .. py:method:: _set_unknown(value) .. py:method:: _get_unknown() .. py:method:: cstruct_children(node, cstruct) .. py:method:: _impl(node, value, callback) .. py:method:: _ser(node, value, options=None) .. py:method:: serialize(node, appstruct, options=None) .. py:method:: _deser(node, value, refs) .. py:method:: deserialize(node, cstruct, refs) .. py:method:: flatten(node, appstruct, prefix='', listitem=False) .. py:method:: unflatten(node, paths, fstruct) .. py:method:: set_value(node, appstruct, path, value) .. py:method:: get_value(node, appstruct, path) .. py:method:: _prepare_save(node, raw_object, saveloc, refs) .. py:method:: _save(node, json_, zipfile_, refs) .. py:method:: save(node, appstruct, zipfile_, refs) .. py:method:: _process_supporting_file(raw_path, zipfile_) raw_path is the filename stored on the object zipfile is an open zipfile.Zipfile in append mode returns the name of the file in the archive .. py:method:: load(node, cstruct, saveloc, refs) .. py:method:: hydrate_json(node, cstruct, saveloc, refs) .. py:method:: _load_supporting_file(filename, saveloc, tmpdir) filename is the name of the file in the zip saveloc can be a folder or open zipfile.ZipFile object if saveloc is a folder and the filename exists inside, this does not return an altered name, nor does it extract to the temporary directory. An altered filename is returned if it cannot find the filename directly or if saveloc is an open zipfile in a temporary directory .. py:method:: _load_json_from_file(fname, saveloc) filename is the name of the file in the zip saveloc can be a folder or open zipfile.ZipFile object .. py:class:: LongLatBounds(*args, **kw) Bases: :py:obj:`colander.SequenceSchema` Used to define bounds on a map .. py:attribute:: bounds .. py:class:: WorldPoint(*arg, **kw) Bases: :py:obj:`LongLat` Used to define reference points. 3D positions (long,lat,z) .. py:attribute:: z .. py:class:: ImageSize(*arg, **kw) Bases: :py:obj:`colander.TupleSchema` Only contains 2D (long, lat) positions .. py:attribute:: width .. py:attribute:: height .. py:function:: now(node, kw) Used by TimeseriesValueSchema - assume it defers the calculation of datetime.datetime.now to when it is called in Schema