CTS2-LE Supported Terminologies: Unterschied zwischen den Versionen

Aus CTS2-LE
Zur Navigation springen Zur Suche springen
Zeile 144: Zeile 144:
 
<p>In this example only the terminologies for age rejection and the exotic one of the ICD as well as the one-time codes of the OPS package will be loaded.</p>
 
<p>In this example only the terminologies for age rejection and the exotic one of the ICD as well as the one-time codes of the OPS package will be loaded.</p>
 
<h2 id="rest-interface">REST interface</h2>
 
<h2 id="rest-interface">REST interface</h2>
<pre>
+
<ul>
GET HOST/WebCts2LE/service/crud/bulk/load-std-terminology
+
<li>HTTP GET <code>&lt;host-cts2le&gt;/WebCts2LE/service/crud/bulk/load-std-terminology</code></li>
</pre>
+
</ul>
 +
<p>where <code>&lt;host-cts2le&gt;</code> is the kubernetes service url for the <code>cts2le</code> container.</p>
 
<h3 id="query-parameters">Query Parameters</h3>
 
<h3 id="query-parameters">Query Parameters</h3>
 
<ul>
 
<ul>
Zeile 153: Zeile 154:
 
</ul>
 
</ul>
 
<h3 id="example-1">Example</h3>
 
<h3 id="example-1">Example</h3>
<pre>
+
<ul>
GET HOST/WebCts2LE/service/crud/bulk/load-std-terminology?directory=/etc/webcts2le/inst/terminologies/loinc&loadSpec=load-spec-2.80.json
+
<li>HTTP GET <code>&lt;host-cts2le&gt;/WebCts2LE/service/crud/bulk/load-std-terminology?directory=/etc/webcts2le/inst/terminologies/loinc&amp;loadSpec=load-spec-2.80.json</code></li>
</pre>
+
</ul>
 
<h3 id="note">Note</h3>
 
<h3 id="note">Note</h3>
 +
<p>Depending on the kubernetes settings, a timeout ca occur. Nevertheless, the loading process is started and can be observed in the container logs. Future versions could introduce a task-concept for such purposes.</p>
 
<p>Afterwards an update of the suggester (used by the navigator) has to be performed:</p>
 
<p>Afterwards an update of the suggester (used by the navigator) has to be performed:</p>
<pre>
+
<ul>
GET HOST/WebCts2LE/service/manage/index/suggester/update
+
<li>HTTP GET <code>&lt;host-cts2le&gt;/WebCts2LE/service/manage/index/suggester/update</code></li>
</pre>
+
</ul>
 
<h2 id="notes">Notes</h2>
 
<h2 id="notes">Notes</h2>
 
<h3 id="removal">Removal</h3>
 
<h3 id="removal">Removal</h3>
 
<p>Removal time is nearly equal load time due to the RDF store. It is recommended that the terminology is not present in the store beforehand.</p>
 
<p>Removal time is nearly equal load time due to the RDF store. It is recommended that the terminology is not present in the store beforehand.</p>
 
<h3 id="loading-time">Loading Time</h3>
 
<h3 id="loading-time">Loading Time</h3>
<p>Due to used RDF quad store technology the loading time (i.e. 'weaving' the RDF-triple knowledge graph based on the flat files) on a notebook (~2 GHz throttled, 16 GB RAM) is:</p>
+
<p>Due to used RDF quad store technology, the loading time (i.e. 'weaving' the RDF-triple knowledge graph based on the flat files) on e.g. openshift is</p>
 
<ul>
 
<ul>
<li>Smomed: ~90 min</li>
+
<li>Smomed: ~15 min</li>
<li>Loinc: ~45 min</li>
+
<li>Loinc: ~9 min</li>
<li>MeSH: ~15 min</li>
+
<li>MeSH: ~7 min</li>
 +
<li>BfArM: ~10 min</li>
 
</ul>
 
</ul>
<p>It is recommended in the context of docker, kubernetes, etc. to define high CPU/RAM resources for the <em>fuseki</em> container to decrease loading time.</p>
+
<p>(CPU ~2.7 GHz (cat /proc/cpuinfo); OS UI POD metrics: ~7.6 GB used memory)</p>
 +
<!-- It is recommended in the context of docker, kubernetes, etc. to define high CPU/RAM resources for the backend container (*cts2le*, *solr*, and especially *fuseki*) to decrease loading time. -->
 
<p>It is a known restriction for quad stores that loading time is high compared to other nosql data stores. On the other hand it offers elaborated functionality based on semantic web technologies. Future versions of CTS2-LE could utilize nosql data stores.</p>
 
<p>It is a known restriction for quad stores that loading time is high compared to other nosql data stores. On the other hand it offers elaborated functionality based on semantic web technologies. Future versions of CTS2-LE could utilize nosql data stores.</p>
<h3 id="database-space">Database Space</h3>
+
<h3 id="disk-space">Disk Space</h3>
<p>Quad stores usually occupies huge space on disk because every quad is indexed. E.g. Snomed has ~18.000.000 quads and the quad store <em>fuseki</em> can occupies ~50 GB after loading. But <em>fuseki</em> offers a compaction of the database to ~8 GB with the call</p>
+
<p>Initially, the following disk space (per container) is occupied for the complete terminology set</p>
 +
<ul>
 +
<li><em>cts2le</em>: ~3 GB</li>
 +
<li><em>fuseki</em>: ~27 GB</li>
 +
<li><em>solr</em>: ~1 GB</li>
 +
</ul>
 +
<p>Deleting and adding the same terminology leads to an increase in resource consumption over time and quad stores usually occupies huge space on disk because every quad is indexed. E.g. Snomed has ~18.000.000 quads. <em>fuseki</em> offers a runtime compaction of the database with the call</p>
 
<pre>
 
<pre>
curl --request POST --url 'http://localhost:3030/$/compact/cts2le?deleteOld=true'
+
curl --request POST --url 'http://fuseki:3030/$/compact/cts2le?deleteOld=true'
 
</pre>
 
</pre>
 
+
<p>For details, please visit <code>https://jena.apache.org/documentation/fuseki2/fuseki-server-protocol.html#compact</code> or contact the Fhg Fokus team.</p>
            <script async src="https://cdn.jsdelivr.net/npm/katex-copytex@latest/dist/katex-copytex.min.js"></script>
 

Version vom 12. Dezember 2025, 14:39 Uhr

Preface

Due to license policies of standard terminology providers we do not make available provider input files. Customers have to download these files from provider sites.

To load a standard terminology the customer has to copy the input files to a dedicated directory (called LD in the following) together with a specification json file (SF). In context of docker, kubernetes etc. a dedicated volume should be used. To avoid inconsistencies, services should not be accessed during load.

Terminology Directory (LD)

Directory LD contains all files required for a specific standard terminology. There should be a separate directory for each terminology (currently Snomed, Loinc, Mesh, and BfArM).

The deployment for Kubernetes provides an extra volume with the path

  • /etc/webcts2le/inst/terminologies

Subdirectories within this path should be used for LD, which is specified when starting the load (see REST interface for loading).

Standard Terminologies

Specification File (SF)

 1 {
 2     "terminologyDesignator": "<regex: 'mesh|loinc|snomed'>",
 3     // usual version string
 4     "version": "<string>",
 5     // group name (used in the navigator)
 6     "groupName": "<string>",
 7     // unique resource id in context of a cts2le instance
 8     "resourceId": "<string>",
 9     // input file paths relative to the given directory <LD>
10     "files": [
11         "<string>"
12         // , ...
13     ]
14 }

Supported Standard Terminologies

 1 {
 2   "terminologyDesignator": "snomed",
 3   "version": "20250515",
 4   "groupName": "snomed-ct",
 5   "resourceId": "Snomed-20250515",
 6   // following files usually are located at 'Snapshot/Terminology' within the snomed zip file
 7   // it is required that the order of the files is stated as below, i.e. 
 8   "defaultLanguage": "de",
 9   "files": [
10     "sct2_Description_Snapshot_GermanyEdition_20250515.txt",
11     "sct2_Concept_Snapshot_GermanyEdition_20250515.txt",
12     "sct2_Relationship_Snapshot_GermanyEdition_20250515.txt",
13     "sct2_StatedRelationship_Snapshot_GermanyEdition_20250515.txt",
14     "sct2_TextDefinition_Snapshot_GermanyEdition_20250515.txt"
15   ]
16 }
 1 {
 2     "terminologyDesignator": "loinc",
 3     "version": "2.80",
 4     "groupName": "loinc-tree",
 5     "resourceId": "Loinc-2.80",
 6     // defines the corr. display name
 7     "defaultLanguage" : "de",
 8     // the following linguistic variants must be exist at directory 'AccessoryFiles/LinguisticVariants',
 9     "linguisticVariants": [
10         {"lang": "de", "file": "Loinc_2.80/AccessoryFiles/LinguisticVariants/deDE15LinguisticVariant.csv"},
11         {"lang": "at", "file": "Loinc_2.80/AccessoryFiles/LinguisticVariants/deAT24LinguisticVariant.csv"}
12     ],
13     "files": [
14         // file 1, 2 must be the hierarchy csv and the table csv, respectively
15         // usually file 1 is at 'AccessoryFiles/MultiAxialHierarchy/' and file 2 at
16         // 'LoincTable/' in the providers zip file, e.g. 'loinc271.zip'
17         "Loinc_2.80/AccessoryFiles/ComponentHierarchyBySystem/ComponentHierarchyBySystem.csv",
18         "Loinc_2.80/LoincTable/Loinc.csv"
19     ]
20 }
1 {
2     "terminologyDesignator": "mesh",
3     "version": "2025",
4     "groupName": "mesh",
5     "resourceId": "Mesh2025",
6     "files": [
7         "desc2025.xml"
8     ]
9 }

BfArM Terminologies

BfArM (Bundesinstitut für Arzneimittel und Medizinprodukte) provides the standard terminologies for Germany. To load these standard terminologies the customer has to create a dedicated directory (called LD in the following) together with a specification json file (SF). In context of docker, kubernetes etc. a dedicated volume should be used.

Packages

The packages downloaded from bfarm must be located in a dedicated directory (e.g. bfarm). The following structure is an example for two packages (ICDGM, OPS).

bfarm
|_ packages
|  |_ bfarm.terminologien.icd10gm-2025.0.0.tar.gz
|  |  |_ package/CodeSystem-icd10gm-agelow-2025.json
|  |  |_ package/CodeSystem-icd10gm-agereject-2025.json
|  |  |_ package/package.json
|  |  |_ ...
|  |_ bfarm.terminologien.ops-2025.0.0.tar.gz
|  |  |_ ...
|_ fhir-packs.jsonc

Specification File (SF)

for BfArM terminologies, the following specification file is used.

1 {
2     "terminologyDesignator": "fhir-package",
3     "canonicalUrlRegex": "<regex>", // optional
4     "packageRegex": "<regex>"
5 }
  • terminologyDesignator

    • has to be set to fhir-package
  • canonicalUrlRegex

    • this filter loads only terminologies whose canonical URL (https://hl7.org/fhir/R4/datatypes.html#canonical) conforms to <regex>. E.g., regex .*(agerejec|agelow).* will only load the terminologies CodeSystem-icd10gm-agereject-2025.json and CodeSystem-icd10gm-agelow-2025.json because its canonical URLs are
      • https: //terminologien.bfarm.de/fhir/CodeSystem/icd10gm-agereject|2025 and
      • https: //terminologien.bfarm.de/fhir/CodeSystem/icd10gm-agelow|2025, respectively.
  • packageRegex

    • this filter loads only packages whose canonical package name conforms to <regex>. The canonical package name ist defined as the form <name>|<version> where name and version are the properties in the package definition file
      • bfarm/packages/bfarm.terminologien.icd10gm-2025.0.0.tar.gz/package/package.json (see section Packages above).
      • E.g., regex .*(icd10gm\\|2025|ops\\|2025).* will only load the ICD and OPS package.

Example

1 {
2     "terminologyDesignator": "fhir-package",
3     "canonicalUrlRegex": ".*(agerejec|exotic|einmalk).*",
4     "packageRegex": ".*(icd10gm\\|2025|ops\\|2025).*"
5 }

In this example only the terminologies for age rejection and the exotic one of the ICD as well as the one-time codes of the OPS package will be loaded.

REST interface

  • HTTP GET <host-cts2le>/WebCts2LE/service/crud/bulk/load-std-terminology

where <host-cts2le> is the kubernetes service url for the cts2le container.

Query Parameters

  • directory: direcory path (LD)
  • loadSpec: path to specification file SF (relative to LD)

Example

  • HTTP GET <host-cts2le>/WebCts2LE/service/crud/bulk/load-std-terminology?directory=/etc/webcts2le/inst/terminologies/loinc&loadSpec=load-spec-2.80.json

Note

Depending on the kubernetes settings, a timeout ca occur. Nevertheless, the loading process is started and can be observed in the container logs. Future versions could introduce a task-concept for such purposes.

Afterwards an update of the suggester (used by the navigator) has to be performed:

  • HTTP GET <host-cts2le>/WebCts2LE/service/manage/index/suggester/update

Notes

Removal

Removal time is nearly equal load time due to the RDF store. It is recommended that the terminology is not present in the store beforehand.

Loading Time

Due to used RDF quad store technology, the loading time (i.e. 'weaving' the RDF-triple knowledge graph based on the flat files) on e.g. openshift is

  • Smomed: ~15 min
  • Loinc: ~9 min
  • MeSH: ~7 min
  • BfArM: ~10 min

(CPU ~2.7 GHz (cat /proc/cpuinfo); OS UI POD metrics: ~7.6 GB used memory)

It is a known restriction for quad stores that loading time is high compared to other nosql data stores. On the other hand it offers elaborated functionality based on semantic web technologies. Future versions of CTS2-LE could utilize nosql data stores.

Disk Space

Initially, the following disk space (per container) is occupied for the complete terminology set

  • cts2le: ~3 GB
  • fuseki: ~27 GB
  • solr: ~1 GB

Deleting and adding the same terminology leads to an increase in resource consumption over time and quad stores usually occupies huge space on disk because every quad is indexed. E.g. Snomed has ~18.000.000 quads. fuseki offers a runtime compaction of the database with the call

curl --request POST --url 'http://fuseki:3030/$/compact/cts2le?deleteOld=true'

For details, please visit https://jena.apache.org/documentation/fuseki2/fuseki-server-protocol.html#compact or contact the Fhg Fokus team.