Database access

HPC-oriented SQLite helpers for reading and writing GO/TS optimizer databases.

Use load_previous_run_results() or SCGODatabaseManager for the common case of reloading minima from completed run_* searches. For low-level access, get_connection() opens a scoped DataConnection with SCGO PRAGMA settings applied.

SCGO Database Module

Designed for HPC use: SQLite on shared filesystems (Lustre, GPFS, NFS-class), batch jobs, and optional multi-process access. WAL mode is off by default. Database discovery uses an in-process registry with a filesystem fallback when the registry has no entries. Prefer job-local scratch for heavy I/O when your site supports it.

scgo.database.get_global_cache(max_size=1000)

Return the global cache singleton.

Return type:

UnifiedCache

scgo.database.close_data_connection(da, log_errors=True)

Safely close a DataConnection object.

Handles the fact that ASE’s SQLite3Database doesn’t have a close() method but does support the context manager protocol (__exit__).

Note

ASE database objects may have their internal SQLite connection invalidated (set to None) in certain conditions (errors, timeouts, external closes). This is a benign state during cleanup and should not produce error messages.

Parameters:

• da (DataConnection | None) – DataConnection object to close (can be None)

• log_errors (bool) – Whether to log errors at debug level (default True)

Return type:

None

Example

>>> da = DataConnection('path/to/db.db')
>>> try:
...     # work with da
... finally:
...     close_data_connection(da)

scgo.database.get_connection(db_path, busy_timeout=30000, wal_mode=False, cache_size_mb=64)

Open and yield an ASE DataConnection (with cleanup on exit).

This is the primary context manager for SCGO database access.

WAL mode is off by default (DELETE journal) for shared/HPC filesystems; pass wal_mode=True on local disks when you need more write concurrency.

Transient sqlite lock errors during open are retried with the same backoff policy used by setup_database().

Parameters:

• db_path (str | Path) – Path to the .db file.

• busy_timeout (int) – SQLite busy timeout in milliseconds (default 30s).

• wal_mode (bool) – If True, apply WAL-related PRAGMAs.

• cache_size_mb (int) – SQLite page cache size hint in MiB.

Return type:

Generator[DataConnection, None, None]

exception scgo.database.DatabaseSetupError

Bases: Exception

Raised when database setup or initialization fails.

exception scgo.database.DatabaseMigrationError

Bases: Exception

Raised when database schema migration fails.

scgo.database.setup_database(output_dir, db_filename, atoms_template, initial_candidate=None, initial_population=None, remove_existing=True, remove_aux_files=False, enable_wal_mode=False, enable_expression_indexes=True, run_id=None)

Create/open an ASE DataConnection for db_filename in output_dir.

Return type:

DataConnection

scgo.database.extract_minima_from_database_file(db_path, run_id, *, require_final=True, persist=False, source_db_relpath=None)

Return minima from db_path annotated with run_id.

Return type:

list[tuple[float, Atoms]]

scgo.database.extract_transition_states_from_database_file(db_path, run_id, *, require_final_unique_ts=True)

Return transition-state rows from db_path with provenance.

Return type:

list[tuple[float, Atoms]]

scgo.database.load_previous_run_results(base_output_dir, db_filename=None, composition=None, current_run_id=None, parallel=True, max_workers=None, prefer_final_unique=True)

Load minima from previous runs for a composition.

Return type:

list[tuple[float, Atoms]]

scgo.database.load_reference_structures(db_glob_pattern, composition=None, max_structures=100, base_dir=None)

Load up to max_structures final minima from databases matching pattern.

Return type:

list[Atoms]

scgo.database.list_discovered_db_paths_with_run(base_dir, *, composition=None, use_cache=True, db_filename=None)

List DB paths via DatabaseDiscovery with run parsed from layout.

Returns tuples (absolute_path, run_id). run_id is empty if the path is not under a recognizable run_* directory.

Return type:

list[tuple[str, str]]

scgo.database.mark_final_minima_in_db(final_minima_info, base_dir, db_paths=None)

Mark final unique minima in database systems.key_value_pairs JSON rows.

Rows are matched by final_id stored in key_value_pairs at relaxed persist time (scgo.utils.helpers.ensure_final_id() via the database adapter’s add_relaxed_step).

Parameters:

• final_minima_info (list[dict]) – List of dicts with keys: ‘energy’ (float), ‘atoms’ (Atoms), ‘rank’ (1-based int), ‘final_written’ (str filepath or filename), ‘final_id’ (str, required)

• base_dir (str | Path) – Base output directory to search for database files (used by discovery)

• db_paths (list[str | Path] | None) – Optional explicit list of database files to search/update

Returns:

summary containing counts, e.g. {“dbs_touched”: int, “rows_updated”: int, “details”: {db_path: rows}}

Return type:

dict

scgo.database.persist_provenance(atoms, run_id=None)

Persist run provenance to atoms.info for discovery.

Return type:

None

class scgo.database.SCGODatabaseManager(base_dir, enable_caching=True, cache_ttl_seconds=300)

Bases: object

Lightweight database manager for SCGO operations.

Provides a high-level cached interface for loading previous run results and reference structures for diversity calculations.

Example

>>> with SCGODatabaseManager(base_dir="output") as manager:
...     refs = manager.load_reference_structures("**/*.db")

clear_cache()

Clear all cached results.

load_previous_results(composition, current_run_id=None, db_filename=None, force_reload=False, prefer_final_unique=True)

Load all minima from previous runs for this composition.

Results are cached for improved performance on repeated calls.

Parameters:

• composition (list[str]) – List of atomic symbols to filter by

• current_run_id (str | None) – Current run ID to exclude from loading

• db_filename (str | None) – Specific database filename to look for (optional)

• force_reload (bool) – Force reload from disk, bypassing cache

• prefer_final_unique (bool) – If True (default), only final_unique_minimum rows are loaded. Set False to include all relaxed structures.

Return type:

list[tuple[float, Atoms]]

Returns:

List of (energy, Atoms) tuples from all previous runs

load_reference_structures(db_glob_pattern, composition=None, max_structures=100, force_reload=False)

Load reference structures for diversity calculation.

Results are cached for improved performance.

Parameters:

• db_glob_pattern (str) – Glob pattern to find database files

• composition (list[str] | None) – Optional composition filter

• max_structures (int) – Maximum number of structures to load

• force_reload (bool) – Force reload from disk, bypassing cache

Return type:

list[Atoms]

Returns:

List of Atoms objects sorted by energy (lowest first)

close()

Release resources held by manager caches.

scgo.database.database_transaction(db, isolation_level='DEFERRED')

Context manager for a transaction.

Yields:

sqlite3.Connection – Raw connection. Commits on success; rolls back on error.

scgo.database.stamp_scgo_database(db_path, *, schema_version=None)

Write scgo_metadata so is_scgo_database() accepts this file.

Used by tests and tools that build SQLite files outside setup_database().

Return type:

None

scgo.database.add_metadata(atoms, run_id=None, generation=None, **extra_metadata)

Add metadata to an Atoms object (stored in atoms.info[‘metadata’]).

Return type:

None

scgo.database.get_metadata(atoms, key, default=None)

Retrieve a metadata value from an Atoms object, or return default.

Order: metadata (canonical), provenance (TS / discovery), then key_value_pairs (ASE DB persisted fields, e.g. raw_score from GA).

Return type:

Any

scgo.database.update_metadata(atoms, **updates)

Update atoms.info['metadata']; mirror raw_score into key_value_pairs (ASE).

Return type:

None

scgo.database.filter_by_metadata(structures, **filters)

Return structures whose metadata match all provided filters.

Return type:

list[Atoms]

class scgo.database.RetryConfig(max_retries=3, initial_delay=0.1, max_delay=5.0, backoff_factor=2.0)

Bases: object

Retry counts and exponential backoff for transient SQLite / I/O errors.

max_retries: int = 3

initial_delay: float = 0.1

max_delay: float = 5.0

backoff_factor: float = 2.0

get_delay(attempt)

Return type:

float

scgo.database.database_retry(operation, config=None, operation_name='database operation', log_level='debug', *, exception_types=None, max_retries=None, initial_delay=None, backoff_factor=None)

Run operation with exponential backoff on transient errors.

When exception_types is None (default), only OperationalError messages classified by is_retryable_error() and any OSError are retried.

When exception_types is set (e.g. HPC_DATABASE_EXCEPTIONS), those exception types are retried without the SQLite message filter — matching the historical retry_with_backoff() behavior used by BH/simple.

Optional max_retries / initial_delay / backoff_factor override fields on config (or PRESET_DEFAULT) for call-site compatibility.

Return type:

Any

scgo.database.retry_transaction(db_connection, operation, config=None, operation_name='transaction', isolation_level='DEFERRED')

Retry a full database transaction on transient lock errors.

Runs operation(conn) inside a fresh database_transaction() on each attempt. Unlike a yielded context manager, this correctly retries when the body or commit raises a retryable OperationalError.

Parameters:

• db_connection – ASE DataConnection (or compatible) for the DB.

• operation (Callable[[Any], Any]) – Callable that receives the SQLite connection and returns a result (may be None).

• config (RetryConfig | None) – Retry/backoff settings (defaults to PRESET_AGGRESSIVE).

• operation_name (str) – Label used in retry log messages.

• isolation_level (str) – SQLite isolation level for each attempt.

Return type:

Any

Returns:

The return value of operation on success.

scgo.database.retry_with_backoff(operation, max_retries=3, initial_delay=0.1, backoff_factor=2.0, exception_types=(<class 'OSError'>, ), operation_name='operation', log_level='debug')

Thin wrapper around database_retry() for backward compatibility.

Return type:

Any

scgo.database.get_registry(base_dir)

Get or create a registry for a base directory.

Parameters:

base_dir (str | Path) – Base directory for the registry

Return type:

DatabaseRegistry

Returns:

DatabaseRegistry instance (cached)

Example

>>> registry = get_registry("output")
>>> db_files = registry.find_databases(composition=["Pt", "Pt"])

scgo.database.clear_registry_cache()

Clear the global registry cache.

Return type:

None