zarr.testing.buffer =================== .. py:module:: zarr.testing.buffer Classes ------- .. autoapisummary:: zarr.testing.buffer.NDBufferUsingTestNDArrayLike zarr.testing.buffer.StoreExpectingTestBuffer zarr.testing.buffer.TestBuffer Module Contents --------------- .. py:class:: NDBufferUsingTestNDArrayLike(array: zarr.core.buffer.core.NDArrayLike) Bases: :py:obj:`zarr.core.buffer.cpu.NDBuffer` Example of a custom NDBuffer that handles MyNDArrayLike .. !! processed by numpydoc !! .. py:method:: all_equal(other: Any, equal_nan: bool = True) -> bool Compare to `other` using np.array_equal. .. !! processed by numpydoc !! .. py:method:: as_ndarray_like() -> NDArrayLike Returns the underlying array (host or device memory) of this buffer This will never copy data. :Returns: The underlying array such as a NumPy or CuPy array. .. .. !! processed by numpydoc !! .. py:method:: as_numpy_array() -> numpy.typing.NDArray[Any] Returns the buffer as a NumPy array (host memory). :Returns: NumPy array of this buffer (might be a data copy) .. .. warning:: Might have to copy data, consider using `.as_ndarray_like()` instead. .. !! processed by numpydoc !! .. py:method:: as_scalar() -> ScalarType Returns the buffer as a scalar value .. !! processed by numpydoc !! .. py:method:: astype(dtype: numpy.typing.DTypeLike, order: Literal['K', 'A', 'C', 'F'] = 'K') -> Self .. py:method:: copy() -> Self .. py:method:: create(*, shape: collections.abc.Iterable[int], dtype: numpy.typing.DTypeLike, order: Literal['C', 'F'] = 'C', fill_value: Any | None = None) -> Self :classmethod: Overwrite `NDBuffer.create` to create an TestNDArrayLike instance .. !! processed by numpydoc !! .. py:method:: empty(shape: zarr.core.common.ChunkCoords, dtype: numpy.typing.DTypeLike, order: Literal['C', 'F'] = 'C') -> Self :classmethod: Create an empty buffer with the given shape, dtype, and order. This method can be faster than ``NDBuffer.create`` because it doesn't have to initialize the memory used by the underlying ndarray-like object. :Parameters: **shape** The shape of the buffer and its underlying ndarray-like object **dtype** The datatype of the buffer and its underlying ndarray-like object **order** Whether to store multi-dimensional data in row-major (C-style) or column-major (Fortran-style) order in memory. :Returns: buffer New buffer representing a new ndarray_like object with empty data. .. seealso:: :obj:`NDBuffer.create` Create a new buffer with some initial fill value. .. !! processed by numpydoc !! .. py:method:: fill(value: Any) -> None .. py:method:: from_ndarray_like(ndarray_like: NDArrayLike) -> Self :classmethod: Create a new buffer of a ndarray-like object :Parameters: **ndarray_like** ndarray-like object :Returns: New buffer representing `ndarray_like` .. .. !! processed by numpydoc !! .. py:method:: from_numpy_array(array_like: numpy.typing.ArrayLike) -> Self :classmethod: Create a new buffer of Numpy array-like object :Parameters: **array_like** Object that can be coerced into a Numpy array :Returns: New buffer representing `array_like` .. .. !! processed by numpydoc !! .. py:method:: reshape(newshape: zarr.core.common.ChunkCoords | Literal[-1]) -> Self .. py:method:: squeeze(axis: tuple[int, Ellipsis]) -> Self .. py:method:: transpose(axes: SupportsIndex | collections.abc.Sequence[SupportsIndex] | None) -> Self .. py:property:: byteorder :type: zarr.codecs.bytes.Endian .. py:property:: dtype :type: numpy.dtype[Any] .. py:property:: shape :type: tuple[int, Ellipsis] .. py:class:: StoreExpectingTestBuffer(store_dict: collections.abc.MutableMapping[str, zarr.core.buffer.Buffer] | None = None, *, read_only: bool = False) Bases: :py:obj:`zarr.storage.MemoryStore` Example of a custom Store that expect MyBuffer for all its non-metadata We assume that keys containing "json" is metadata .. !! processed by numpydoc !! .. py:method:: clear() -> None :async: Clear the store. Remove all keys and values from the store. .. !! processed by numpydoc !! .. py:method:: close() -> None Close the store. .. !! processed by numpydoc !! .. py:method:: delete(key: str) -> None :async: Remove a key from the store :Parameters: **key** : str .. .. !! processed by numpydoc !! .. py:method:: delete_dir(prefix: str) -> None :async: Remove all keys and prefixes in the store that begin with a given prefix. .. !! processed by numpydoc !! .. py:method:: exists(key: str) -> bool :async: Check if a key exists in the store. :Parameters: **key** : str .. :Returns: bool .. .. !! processed by numpydoc !! .. py:method:: get(key: str, prototype: zarr.core.buffer.BufferPrototype, byte_range: tuple[int, int | None] | None = None) -> zarr.core.buffer.Buffer | None :async: Retrieve the value associated with a given key. :Parameters: **key** : str .. **prototype** : BufferPrototype The prototype of the output buffer. Stores may support a default buffer prototype. **byte_range** : ByteRequest, optional ByteRequest may be one of the following. If not provided, all data associated with the key is retrieved. - RangeByteRequest(int, int): Request a specific range of bytes in the form (start, end). The end is exclusive. If the given range is zero-length or starts after the end of the object, an error will be returned. Additionally, if the range ends after the end of the object, the entire remainder of the object will be returned. Otherwise, the exact requested range will be returned. - OffsetByteRequest(int): Request all bytes starting from a given byte offset. This is equivalent to bytes={int}- as an HTTP header. - SuffixByteRequest(int): Request the last int bytes. Note that here, int is the size of the request, not the byte offset. This is equivalent to bytes=-{int} as an HTTP header. :Returns: Buffer .. .. !! processed by numpydoc !! .. py:method:: get_partial_values(prototype: zarr.core.buffer.BufferPrototype, key_ranges: collections.abc.Iterable[tuple[str, zarr.abc.store.ByteRequest | None]]) -> list[zarr.core.buffer.Buffer | None] :async: Retrieve possibly partial values from given key_ranges. :Parameters: **prototype** : BufferPrototype The prototype of the output buffer. Stores may support a default buffer prototype. **key_ranges** : Iterable[tuple[str, tuple[int | None, int | None]]] Ordered set of key, range pairs, a key may occur multiple times with different ranges :Returns: list of values, in the order of the key_ranges, may contain null/none for missing keys .. .. !! processed by numpydoc !! .. py:method:: getsize(key: str) -> int :async: Return the size, in bytes, of a value in a Store. :Parameters: **key** : str .. :Returns: **nbytes** : int The size of the value (in bytes). :Raises: FileNotFoundError When the given key does not exist in the store. .. !! processed by numpydoc !! .. py:method:: getsize_prefix(prefix: str) -> int :async: Return the size, in bytes, of all values under a prefix. :Parameters: **prefix** : str The prefix of the directory to measure. :Returns: **nbytes** : int The sum of the sizes of the values in the directory (in bytes). .. seealso:: :obj:`zarr.Array.nbytes_stored` .. :obj:`Store.getsize` .. .. rubric:: Notes ``getsize_prefix`` is just provided as a potentially faster alternative to listing all the keys under a prefix calling :meth:`Store.getsize` on each. In general, ``prefix`` should be the path of an Array or Group in the Store. Implementations may differ on the behavior when some other ``prefix`` is provided. .. !! processed by numpydoc !! .. py:method:: is_empty(prefix: str) -> bool :async: Check if the directory is empty. :Parameters: **prefix** : str Prefix of keys to check. :Returns: bool True if the store is empty, False otherwise. .. !! processed by numpydoc !! .. py:method:: list() -> collections.abc.AsyncIterator[str] :async: Retrieve all keys in the store. :Returns: AsyncIterator[str] .. .. !! processed by numpydoc !! .. py:method:: list_dir(prefix: str) -> collections.abc.AsyncIterator[str] :async: Retrieve all keys and prefixes with a given prefix and which do not contain the character “/” after the given prefix. :Parameters: **prefix** : str .. :Returns: AsyncIterator[str] .. .. !! processed by numpydoc !! .. py:method:: list_prefix(prefix: str) -> collections.abc.AsyncIterator[str] :async: Retrieve all keys in the store that begin with a given prefix. Keys are returned relative to the root of the store. :Parameters: **prefix** : str .. :Returns: AsyncIterator[str] .. .. !! processed by numpydoc !! .. py:method:: open(*args: Any, **kwargs: Any) -> Self :classmethod: :async: Create and open the store. :Parameters: **\*args** : Any Positional arguments to pass to the store constructor. **\*\*kwargs** : Any Keyword arguments to pass to the store constructor. :Returns: Store The opened store instance. .. !! processed by numpydoc !! .. py:method:: set(key: str, value: zarr.core.buffer.Buffer, byte_range: tuple[int, int] | None = None) -> None :async: Store a (key, value) pair. :Parameters: **key** : str .. **value** : Buffer .. .. !! processed by numpydoc !! .. py:method:: set_if_not_exists(key: str, value: zarr.core.buffer.Buffer) -> None :async: Store a key to ``value`` if the key is not already present. :Parameters: **key** : str .. **value** : Buffer .. .. !! processed by numpydoc !! .. py:method:: set_partial_values(key_start_values: collections.abc.Iterable[tuple[str, int, bytes | bytearray | memoryview[int]]]) -> None :abstractmethod: :async: Store values at a given key, starting at byte range_start. :Parameters: **key_start_values** : list[tuple[str, int, BytesLike]] set of key, range_start, values triples, a key may occur multiple times with different range_starts, range_starts (considering the length of the respective values) must not specify overlapping ranges for the same key .. !! processed by numpydoc !! .. py:method:: with_read_only(read_only: bool = False) -> MemoryStore Return a new store with a new read_only setting. The new store points to the same location with the specified new read_only state. The returned Store is not automatically opened, and this store is not automatically closed. :Parameters: **read_only** If True, the store will be created in read-only mode. Defaults to False. :Returns: A new store of the same type with the new read only attribute. .. .. !! processed by numpydoc !! .. py:property:: read_only :type: bool Is the store read-only? .. !! processed by numpydoc !! .. py:property:: supports_consolidated_metadata :type: bool Does the store support consolidated metadata?. If it doesn't an error will be raised on requests to consolidate the metadata. Returning `False` can be useful for stores which implement their own consolidation mechanism outside of the zarr-python implementation. .. !! processed by numpydoc !! .. py:attribute:: supports_deletes :type: bool :value: True Does the store support deletes? .. !! processed by numpydoc !! .. py:attribute:: supports_listing :type: bool :value: True Does the store support listing? .. !! processed by numpydoc !! .. py:attribute:: supports_partial_writes :type: bool :value: True Does the store support partial writes? .. !! processed by numpydoc !! .. py:attribute:: supports_writes :type: bool :value: True Does the store support writes? .. !! processed by numpydoc !! .. py:class:: TestBuffer(array_like: zarr.core.buffer.core.ArrayLike) Bases: :py:obj:`zarr.core.buffer.cpu.Buffer` Example of a custom Buffer that handles ArrayLike .. !! processed by numpydoc !! .. py:method:: as_array_like() -> ArrayLike Returns the underlying array (host or device memory) of this buffer This will never copy data. :Returns: The underlying 1d array such as a NumPy or CuPy array. .. .. !! processed by numpydoc !! .. py:method:: as_buffer_like() -> zarr.core.common.BytesLike Returns the buffer as an object that implements the Python buffer protocol. :Returns: An object that implements the Python buffer protocol .. .. rubric:: Notes Might have to copy data, since the implementation uses `.as_numpy_array()`. .. !! processed by numpydoc !! .. py:method:: as_numpy_array() -> numpy.typing.NDArray[Any] Returns the buffer as a NumPy array (host memory). :Returns: NumPy array of this buffer (might be a data copy) .. .. rubric:: Notes Might have to copy data, consider using `.as_array_like()` instead. .. !! processed by numpydoc !! .. py:method:: create_zero_length() -> Self :classmethod: Create an empty buffer with length zero :Returns: New empty 0-length buffer .. .. !! processed by numpydoc !! .. py:method:: from_array_like(array_like: ArrayLike) -> Self :classmethod: Create a new buffer of an array-like object :Parameters: **array_like** array-like object that must be 1-dim, contiguous, and byte dtype. :Returns: New buffer representing `array_like` .. .. !! processed by numpydoc !! .. py:method:: from_buffer(buffer: zarr.core.buffer.core.Buffer) -> Self :classmethod: Create a new buffer of an existing Buffer This is useful if you want to ensure that an existing buffer is of the correct subclass of Buffer. E.g., MemoryStore uses this to return a buffer instance of the subclass specified by its BufferPrototype argument. Typically, this only copies data if the data has to be moved between memory types, such as from host to device memory. :Parameters: **buffer** buffer object. :Returns: A new buffer representing the content of the input buffer .. .. rubric:: Notes Subclasses of `Buffer` must override this method to implement more optimal conversions that avoid copies where possible .. !! processed by numpydoc !! .. py:method:: from_bytes(bytes_like: zarr.core.common.BytesLike) -> Self :classmethod: Create a new buffer of a bytes-like object (host memory) :Parameters: **bytes_like** bytes-like object :Returns: New buffer representing `bytes_like` .. .. !! processed by numpydoc !! .. py:method:: to_bytes() -> bytes Returns the buffer as `bytes` (host memory). :Returns: `bytes` of this buffer (data copy) .. .. warning:: Will always copy data, only use this method for small buffers such as metadata buffers. If possible, use `.as_numpy_array()` or `.as_array_like()` instead. .. !! processed by numpydoc !!