Experimental

Experimental functions with tentative APIs. More...

Functions
Qureg	createQuregFromFile (const char *fn)
int	getQuESTNumGpuThreadsPerBlock ()
void	initCustomMpiCommQuESTEnv (MPI_Comm questComm, int useGpuAccel, int useMultithread)
void	initCustomMpiQuESTEnv (int useDistrib, bool userOwnsMpi, int useGpuAccel, int useMultithread)
void	saveQuregToFile (Qureg qureg, const char *fn)
void	setQuESTNumGpuThreadsPerBlock (int numThreadsPerBlock)

Detailed Description

Experimental functions with tentative APIs.

Function Documentation

createQuregFromFile()

Qureg createQuregFromFile

(

const char *

fn

)

Creates a new Qureg from a file (or folder) previously created by saveQuregToFile(), with automatically chosen deployments (independent of those used when the file was saved), and populates the Qureg with the saved amplitudes.

The chosen deployments are identical to those chosen by createQureg() and createDensityQureg().

Note

The number of distributed nodes chosen by the autodeployer must agree with the number of nodes of the originally saved Qureg, else a error is thrown. Therefore, the number of MPI processes calling these functions cannot be changed between saveQuregToFile() and createQuregFromFile(), unless the Qureg was non-distributed in both settings.

Important

This function is only callable when QuEST is compiled with CMake option QUEST_ENABLE_ADIOS2=1.

Parameters

[in]

fn

the file (or folder) path previously created by saveQuregToFile().

Returns

A new Qureg instance matching the saved dimension and amplitudes.

Exceptions

error

if QuEST was not compiled with CMake option QUEST_ENABLE_ADIOS2=1. if fn cannot be read (since, for example, it does not exist). if the precision of the saved Qureg differs from the current QuEST precision. if the number of distributed nodes of the saved Qureg differs from the autodeployer's chosen number. if the recorded Qureg dimensions would overflow the qindex type. if the recorded total Qureg memory would overflow the size_t type. if the system contains insufficient RAM (or VRAM) to store the Qureg in any deployment. if any Qureg memory allocation unexpectedly fails.

See also

• saveQuregToFile() to create a file readable by this function.

Author

Ashmit JaiSarita Gupta

Tyson Jones (input validation)

Definition at line 240 of file experimental.cpp.

                                          {
    validate_adios2IsCompiled(__func__);
 
#if QUEST_COMPILE_ADIOS2
 
    // pedantic but safe - don't let ADIOS2 start reading while other processes are working
    if (comm_isActive())
        comm_sync();
 
    // make ADIOS2 MPI-aware even when the subsequently-loaded Qureg is
    // auto-deployed to be non-distributed; every process will safely
    // parse the file and independently update its Qureg copy
    bool giveAdiosMpi = comm_isActive();
 
    // gratuitously re-create ADIOS2 at every call, for simplicity (occluded by IO)
    adios2::ADIOS adios = createAdios(giveAdiosMpi);
    adios2::IO io = adios.DeclareIO("QuESTQuregLoad");
 
    // use BP5 specifically to avoid non-root-hangs upon rank exceptions
    io.SetEngine("BP5");
 
    // attempt to open the file, and prepare to parse
    adios2::Engine engine; // default ctor
    bool success = false;
    try {
        engine = io.Open(fn, adios2::Mode::ReadRandomAccess);
        success = true;
    } catch (...) {}
    validate_adiosCanOpenFileOnAllNodes(success, fn, __func__);
 
    // check that the file contains the expected variables
    auto vNumQubits  = io.InquireVariable<int>("numQubits");
    auto vNumNodes   = io.InquireVariable<int>("numNodes");
    auto vIsDensMatr = io.InquireVariable<int>("isDensityMatrix");
    auto vQrealBytes = io.InquireVariable<size_t>("qrealBytes");
    auto vAmpComponents = io.InquireVariable<qreal>("ampComponents");
    bool areAllVarsPresent = vNumQubits && vNumNodes && vIsDensMatr && vQrealBytes && vAmpComponents;
    validate_adiosFileContainsFieldsOnAllNodes(areAllVarsPresent, __func__);
 
    // read dimension + precision metadata first, so we can size the new Qureg
    int numQubits = 0;
    int numNodes = 0;
    int isDensMatr = 0;
    size_t fileQrealBytes = 0;
    success = false;
    try {
        engine.Get(vNumQubits,  numQubits);
        engine.Get(vNumNodes,   numNodes);
        engine.Get(vIsDensMatr, isDensMatr);
        engine.Get(vQrealBytes, fileQrealBytes);
        engine.PerformGets();
        success = true;
    } catch (...) {}
    validate_adiosCanReadFileOnAllNodes(success, fn, __func__);
 
    // check the amps are of the expected precision, and so are parsable
    validate_newQuregFileMatchesPrecision(fileQrealBytes, __func__);
 
    // attempt to create a matching-dimension Qureg with automatically chosen deployments
    Qureg qureg = validateAndCreateCustomQureg(numQubits, isDensMatr, 
        modeflag::USE_AUTO, modeflag::USE_AUTO, modeflag::USE_AUTO, __func__);
 
    // auto-distribution MUST match checkpointed distribution (pre-free to avoid leak)
    if (qureg.numNodes != numNodes)
        destroyQureg(qureg);
    validate_newQuregNumNodesMatchesSavedFile(numNodes, qureg.numNodes, comm_getNumNodes(), numQubits, isDensMatr, __func__);
 
    // read only this node's slice of the global amplitude array into its buffer
    // (guaranteed not to overflow by above validateAndCreateCustomQureg validation)
    qindex localReals = 2 * qureg.numAmpsPerNode;
    qindex startReal  = 2 * ((qindex) qureg.rank) * qureg.numAmpsPerNode;
    vAmpComponents.SetSelection({ { (size_t) startReal }, { (size_t) localReals } });
    success = false;
    try {
        engine.Get(vAmpComponents, reinterpret_cast<qreal*>(qureg.cpuAmps)); // immediate; PerformGets redundant
        success = true;
    } catch (...) {}
    validate_adiosCanReadFileOnAllNodes(success, fn, __func__);
 
    // complete ADIOS2 work
    success = false;
    try {
        engine.Close();
        success = true;
    } catch (...) {}
    validate_adiosCanReadFileOnAllNodes(success, fn, __func__);
 
    // propagate the restored CPU amplitudes to the GPU, if deployed
    if (qureg.isGpuAccelerated)
        gpu_copyCpuToGpu(qureg);
 
    return qureg;
#else
    // unreachable: the validation above always throws in non-checkpointing builds
    return Qureg{};
#endif
}

Referenced by createQuregFromFile().

getQuESTNumGpuThreadsPerBlock()

int getQuESTNumGpuThreadsPerBlock

(

)

Note

Documentation for this function or struct is under construction!

Author

Oliver Brown

Definition at line 137 of file experimental.cpp.

                                    {
    validate_envIsInit(__func__);
    
    return gpu_getNumThreadsPerBlock();
}

Referenced by TEST_CASE().

initCustomMpiCommQuESTEnv()

void initCustomMpiCommQuESTEnv	(	MPI_Comm	questComm,
		int	useGpuAccel,
		int	useMultithread )

Note

Documentation for this function or struct is under construction!

Advanced initialiser which allows the user to provide an MPI communicator for QuEST to use. Use of this initialiser implies userOwnsMpi = true, (exposed by initCustomMpiQuESTEnv) and therefore that they have already initialised MPI, and they will call MPI_Finalize at the appropriate time.

The user-provided MPI communicator undergoes the same validation procedure as any that QuEST would use, and so must contain a power-of-2 number of processes.

Important

This function is only compiled and exposed when macro QUEST_COMPILE_SUBCOMM is 1, as is defined when providing CMake option QUEST_ENABLE_SUBCOMM during building.

Author

Oliver Brown

Definition at line 113 of file experimental.cpp.

                                                                                            {
 
    // useDistrib and userOwnsMpi are implied by the user of this initialiser
    const int useDistrib = 1;
    const bool userOwnsMpi = true;
 
    // pre-validate that we are able to set the MPI communicator
    validate_mpiInitStatus(useDistrib, userOwnsMpi, __func__);
    validate_mpiSubCommIsNonNull(userQuestComm != MPI_COMM_NULL, __func__);
 
    // avoid re-setting the MPI comm (to avoid an internal error), which happens
    // if a user illegally re-calls this function, which will be subsequently
    // caught by the validation in validateAndInitCustomQuESTEnv() below
    if (!comm_isActive()) {
        bool success = comm_setMpiComm(userQuestComm, userOwnsMpi);
        validate_mpiSubCommSetSucceeded(success, __func__);
    }
 
    // perform remaining validation (some is harmlessly repeated) and init QuEST env
    validateAndInitCustomQuESTEnv(useDistrib, userOwnsMpi, useGpuAccel, useMultithread, __func__);
}

initCustomMpiQuESTEnv()

void initCustomMpiQuESTEnv	(	int	useDistrib,
		bool	userOwnsMpi,
		int	useGpuAccel,
		int	useMultithread )

Note

Documentation for this function or struct is under construction!

Advanced initialiser which lets the user positively declare that they take responsibility for MPI. This means we assume they have called MPI_Init, and that they will call MPI_Finalize.

Author

Oliver Brown

Definition at line 107 of file experimental.cpp.

                                                                                                  {
    validateAndInitCustomQuESTEnv(useDistrib, userOwnsMpi, useGpuAccel, useMultithread, __func__);
}

saveQuregToFile()

void saveQuregToFile	(	Qureg	qureg,
		const char *	fn )

Writes the contents of qureg to the file (or folder) fn, so that it may later be restored with createQuregFromFile(), potentially in another process.

The output records the qureg dimension (number of qubits and whether it is a density matrix), the amplitude precision, the Qureg's distribution, and the Qureg's full set of amplitudes. Other deployment information, such as whether the Qureg is multithreaded or GPU-accelerated, is not recorded.

There is no particular file extension or folder name suffix required, though since saving is performed with ADIOS2, a suffix of .bp is conventional.

Attention

Specifying fn equal to an existing directory or file will cause erasure and overwriting of its contents. It is especially dangerous to pass fn equal to a system directory, such as / on Unix, and may cause system corruption.

Important

This function is only callable when QuEST is compiled with CMake option QUEST_ENABLE_ADIOS2=1.

Parameters

[in]	qureg	the Qureg to write to disk.
[in]	fn	the output file (or folder) path.

Exceptions

error

if qureg is uninitialised. if QuEST was not compiled with CMake option QUEST_ENABLE_ADIOS2=1. if opening or writing to fn fails.

See also

• createQuregFromFile() to restore a Qureg saved by this function.

Author

Ashmit JaiSarita Gupta

Definition at line 155 of file experimental.cpp.

                                                  {
    validate_adios2IsCompiled(__func__);
    validate_quregFields(qureg, __func__);
    
    (void) fn; // suppress unused warning
 
#if QUEST_COMPILE_ADIOS2
 
    // When the QuEST env is distributed, but the given Qureg is not (and is instead
    // duplicated upon every node), we should permit only a single process (the root)
    // to use ADIOS2 to write to the (assumably, shared) filesystem. Note that non-root
    // nodes must not exit; they need to participate in validation syncs
    bool shouldSkipAdios = (! qureg.isDistributed) && (comm_getRank() > ROOT_RANK);
 
    // pedantic but safe - don't let ADIOS2 start reading amps prematurely
    if (qureg.isDistributed)
        comm_sync();
    
    // TODO:
    // We can optimise in GPU settings by giving ADIOS2 the device memory
    // pointers; but for now, we simply stage into CPU memory first
    if (qureg.isGpuAccelerated)
        gpu_copyGpuToCpu(qureg);
 
    // gratuitously re-create ADIOS2 at every call, for simplicity (occluded by IO)
    adios2::ADIOS adios = createAdios(qureg.isDistributed);
    adios2::IO io = adios.DeclareIO("QuESTQuregSave");
 
    // use BP5 specifically to avoid non-root-hangs upon rank exceptions
    io.SetEngine("BP5");
 
    // attempt to open the file
    adios2::Engine engine; // default ctor
    bool success = false;
    try {
        if (!shouldSkipAdios)
            engine = io.Open(fn, adios2::Mode::Write);
        success = true;
    } catch (...) {}
    validate_adiosCanOpenFileOnAllNodes(success, fn, __func__);
 
    // global single-value metadata; we deliberately record only the dimension
    // and precision, never incidental deployment fields (the loader chooses its
    // own deployment) nor derivable fields (like numAmps)
    adios2::Variable<int> vNumQubits  = io.DefineVariable<int>("numQubits");
    adios2::Variable<int> vNumNodes   = io.DefineVariable<int>("numNodes");
    adios2::Variable<int> vIsDensMatr = io.DefineVariable<int>("isDensityMatrix");
    adios2::Variable<size_t> vQrealBytes = io.DefineVariable<size_t>("qrealBytes"); // also encodes precision
 
    // amplitudes are stored as interleaved (real, imag) reals to stay agnostic
    // to precision and to ADIOS2's complex-type support; each node writes only
    // its local slice into the global array, avoiding excessive memory use
    // (these scalars are guaranteed not to overflow by createQureg validation)
    qindex globalReals = 2 * qureg.numAmps;
    qindex localReals  = 2 * qureg.numAmpsPerNode;
    qindex startReal   = localReals * qureg.rank;
    adios2::Variable<qreal> vAmpComponents = io.DefineVariable<qreal>(
        "ampComponents",
        { (size_t) globalReals },
        { (size_t) startReal },
        { (size_t) localReals });
 
    // attempt to write to file
    success = false;
    try {
        if (!shouldSkipAdios) {
            engine.Put(vNumQubits,  qureg.numQubits);
            engine.Put(vNumNodes,   qureg.numNodes);
            engine.Put(vIsDensMatr, qureg.isDensityMatrix);
            engine.Put(vQrealBytes, sizeof(qreal));
            engine.Put(vAmpComponents, reinterpret_cast<qreal*>(qureg.cpuAmps));
            engine.Close();
        }
        success = true;
    } catch (...) {}
    validate_adiosCanWriteToFileOnAllNodes(success, fn, __func__);
 
    // prevent any process from continuing until ADIOS2 is fully finished
    if (qureg.isDistributed)
        comm_sync();
 
#endif
}

Referenced by saveQuregToFile().

setQuESTNumGpuThreadsPerBlock()

void setQuESTNumGpuThreadsPerBlock

(

int

numThreadsPerBlock

)

Overrides the number of CUDA threads per block (or blockDim) used by QuEST's GPU-accelerated backend.

This changes the GPU parallelisation granularity and can affect performance, and is useful for performance tuning or diagnostics. Before this function is called, QuEST will use the number as specified by the environment variable QUEST_DEFAULT_NUM_GPU_THREADS_PER_BLOCK, if defined. Otherwise, it will use the value specified by the CMake/compile option of the same name, which itself presently defaults to 128. After this function is called, QuEST will adopt numThreadsPerBlock for the remainder of execution, or until this function is called again.

Practical values of numThreadsPerBlock can vary with the simulation size, the user's GPU hardware, and whether it is NVIDIA or AMD, which have respective warp sizes of 32 and 64.

Note

This function has no effect when QuEST is not deployed with GPU-acceleration enabled.

Parameters

[in]

numThreadsPerBlock

the new block size.

Exceptions

error

if the QuESTEnv is not initialised. if numThreadsPerBlock is negative. if numThreadsPerBlock is not a multiple of the GPU warp size. if numThreadsPerBlock exceeds the maximum blockDim imposed by the GPU hardware.

See also

• QUEST_DEFAULT_NUM_GPU_THREADS_PER_BLOCK

Author

Oliver Brown

Tyson Jones

Definition at line 144 of file experimental.cpp.

                                               {
    validate_envIsInit(__func__);
 
    // validation messages and queries depend upon GPU usage
    bool gpuIsActive = getQuESTEnv().isGpuAccelerated;
    validate_numGpuThreadsPerBlock(numTPB, gpuIsActive, __func__);
 
    gpu_setNumThreadsPerBlock(numTPB);
}

Referenced by TEST_CASE().