Actions for repository managers: Correct labelling of languages

May 4, 2023

When the language of the resource is correctly attributed, it allows discovery and indexing services to properly process and parse the text. Indexing involves text analysis practices such as stemming, lemmatization (grouping together the inflected forms of a word so they can be analysed as a single item), and the appropriate treatment of stop-words, all of which are language specific. Including the language tag enables information seekers, aggregators, and other discovery services to correctly identify the language of the full text and treat items accordingly.

Declare the language of the resource at the item level

¶ 20 Leave a comment on paragraph 20 0 Declaring the primary language of the document is considered mandatory. The language metadata must be encoded using the ISO-639 language code.

¶ 21 Leave a comment on paragraph 21 1 If the document has only one language, language metadata identifies the primary language of the resource. Attribution of the primary language of the resource must be done at the item level.

Example 1: language in simple Dublin Core XML with ISO-639-1 encoding

 <dc:language>en</dc:language>

Example 2: language in MODS with ISO 639-2 encoding

<language>
<languageTerm authority="iso639-2b" type="code" uthorityURI="http://id.loc.gov/vocabulary/iso639-2" valueURI="http://id.loc.gov/vocabulary/iso639-2/eng">eng</languageTerm>
</language>

If the whole document or parts of the documents contains more than one language, the language metadata should be repeated to mention each language.

Example 3: bilingual (french/english) document in simple Dublin Core XML with ISO-639-1 encoding

<dc:language>en</dc:language>
<dc:language>fr</dc:language>

Example 4: bilingual (french/english) document in MODS with ISO 639-2 encoding

<language>
<languageTerm authority="iso639-2b" type="code" uthorityURI="http://id.loc.gov/vocabulary/iso639-2" valueURI="http://id.loc.gov/vocabulary/iso639-2/eng">eng</languageTerm>
</language>
<language>
<languageTerm authority="iso639-2b" type="code" uthorityURI="http://id.loc.gov/vocabulary/iso639-2" valueURI="http://id.loc.gov/vocabulary/iso639-2/fre">fre</languageTerm>
</language>

See more implementation examples following metadata standards/guidelines in the Appendix 2.

Declare the language of the metadata (xml:lang attribute)

¶ 22 Leave a comment on paragraph 22 2 Use the xml:lang attribute to indicate the language of the metadata field. Because of the cardinality [0-n], the xml:lang attribute could describe the same element in different languages, so this would be more accurate than the dc:language element.

¶ 23 Leave a comment on paragraph 23 0 Declare the language of the metadata even in English. It seems like an additional effort, but it’s worth it, aggregators can’t assume that.

¶ 24 Leave a comment on paragraph 24 2 The xml:lang attribute/subproperty is described at https://www.w3.org/TR/xml/#sec-lang-tag . The values of the attribute are language identifiers as defined by [IETF BCP 47], Tags for the Identification of Languages.

How to attribute language when there is more than one language in the metadata fields

Use of the xml:lang attribute to indicate the language of the metadata field.

<datacite:titles>
<datacite:title xml:lang="en">Open Access</datacite:title>
<datacite:title xml:lang="pl">Otwarty Dostęp</datacite:title>
</datacite:titles>

<dc:title xml:lang="en">Open Access</dc:title>
<dc:title xml:lang="fr">Libre Accès</dc:title>

See more implementation examples following metadata standards/guidelines in the Appendix 3.

Use two-letter language codes whenever they are available, and three-letter codes if necessary (ISO 639)

¶ 25 Leave a comment on paragraph 25 4 IANA recommends using two-letter codes whenever they are available, and three-letter codes if necessary (e.g. if no two letter code exists):
http://www.iana.org/assignments/language-subtag-registry/language-subtag-registry and https://en.wikipedia.org/wiki/IETF_language_tag.

According to the “Internet Official Protocol Standards” IETF RFC 1766 : Tags for the Identification of Languages: “The language tag is composed of 1 or more parts: A primary language tag and a (possibly empty) series of subtags.

In the primary language tag all 2-letter tags are interpreted according to ISO standard 639, “Code for the representation of names of languages” [ISO 639].
The information in the subtag may for instance be:

Country identification, such as en-US (this usage is described in ISO 639)

Dialect or variant information, such as no-nynorsk or en-cockney

Languages not listed in ISO 639 that are not variants of any listed language, which can be registered with the i-prefix, such as i-silbo

Script variations, such as az-arabic and az-cyrillic”

Using language codes can also be practical for historical or non-official languages (e.g. Latin, Walloon, etc.). Examples in Walloon:

https://orbi.uliege.be/handle/2268/28421
https://orbi.uliege.be/handle/2268/28419

See more about ISO 639-1, ISO 639-2 and ISO 639-3 and language tags in Appendix 4.

Implementing recommendations in repository software platforms

Dataverse

¶ 26 Leave a comment on paragraph 26 0 Dataverse is the open source data repository developed by IQSS of Harvard University. A strong Dataverse community is helping to improve the basic functionality and develop it further. DANS-KNAW delivered production ready (Docker/k8s) Dataverse repository for the European Open Science Cloud (EOSC) communities CESSDA, CLARIN and DARIAH. To address the heterogeneous and multilingual datasets integration challenges, DANS-KNAW introduced external controlled vocabularies support (CESSDA Metadata Model connected to Skosmos framework; support for CLARIN Component MetaData Infrastructure and the European Language Social Science Thesaurus (ELSST) hosted by CESSDA and ODISSEI in Skosmos – CESSDA has an updated version with more language properties).

DSpace

¶ 27 Leave a comment on paragraph 27 1 DSpace 7 allows using a language attribute for any metadata you want – a two-character ISO in the dropdown menu, but in the field you can write anything and there are different variations. E.g. you can add more than one language option for an item – for several fields, but you can’t specify on which field you are referring to. For earlier versions of DSpace there was a need to find workarounds for this. See Appendix 5 on how to fix language code inconsistencies in repositories running on previous versions of DSpace.

¶ 28 Leave a comment on paragraph 28 0 However, there is no exposure of the language of metadata in OAI-PMH and it’s a request to software developers.

Multiple repository interface languages

¶ 29 Leave a comment on paragraph 29 1 If the repository software supports multiple interface languages, it is recommended to set up the user interface in the native language(s) of the target group, along with that in English.

¶ 30 Leave a comment on paragraph 30 0 DSpace provides support for multiple interface languages. The text displayed on the interface is called “messages” and the messages files (language packs) are contributed and managed by the community outside the core DSpace project to allow more regular updates and releases. Users can modify community translations or create their own and commit them to the dspace-api-lang project on Github. Apart from messages, it is possible to localize other elements, such as help pages, input forms and email templates. Instructions on how to enable the interface in multiple languages is available in the DSpace documentation. DSpace 7 makes a step forward towards facilitating UI translations: https://wiki.lyrasis.org/pages/viewpage.a:

¶ 31 Leave a comment on paragraph 31 0 Dataverse supports multilingual user interfaces and relies on community translations done by volunteers. Major progress towards creating a directory of language packs was made within the Social Sciences and Humanities Open Cloud (SSHOC) project and the online tool Weblate was designed to facilitate new translations. A user guide for Weblate is also available: https://doi.org/10.5281/zenodo.4807371.

EPrints

¶ 32 Leave a comment on paragraph 32 0 EPrints provides support for multiple interface languages, using language-specific folders of “phrases” and other files. By default, EPrints only comes packaged with English language phrases, but the community has shared many translations in the EPrints Bazaar and on EPrints Files. EPrints uses the two letter ISO language standard to specify sub-directories of phrases and other types of language specific directories, for example:

¶ 33 Leave a comment on paragraph 33 0
lib/lang/en/phrases/
lib/lang/fr/static/
lib/lang/de/templates/

¶ 34 Leave a comment on paragraph 34 0 EPrints subject metadata is designed to accommodate multilingual labels, so subject labels can be displayed according to what language the user has set for the interface.

¶ 35 Leave a comment on paragraph 35 0 EPrints is designed to default to English phrases; if it has missing phrases for another declared interface language, it will use the English language phrases until the missing phrases are added. There is a technical wiki page about translations but it may be out of date as it has only been edited a few times in the last few years.

¶ 36 Leave a comment on paragraph 36 0 EPrints can be extended to declare language information at the item or file level but this is not in place on EPrints by default. Similarly EPrints XML export plugins, embedded metadata and OAI-PMH interface code could be extended to define xml:lang attributes but it does not do this by default.

OSF – Open Science Framework

¶ 37 Leave a comment on paragraph 37 0 New metadata enhancements on OSF for all OSF Projects, Registrations, and Preprints now includes the language of materials, more details in New OSF Metadata to Support Data Sharing Policy Compliance.

TIND IR

https://www.tind.io/ir

¶ 38 Leave a comment on paragraph 38 1 The TIND IR is a MARC-based repository. That means that the easiest way to include information about multilingual content is through the 041 field and relevant subfields (https://www.loc.gov/marc/bibliographic/bd041.html). While the language codes used for cataloguing (https://www.loc.gov/marc/languages/language_name.html) do not conform to the recommendation of this group, the processes and details of repository entries are more flexible and should probably instead use the language code methods described in this recommendation. The use of subfields allows for granular declaration of item language, summary language, table of contents language, and more.

The XML might look something like the following:

<datafield tag="041" ind1="0" ind2=" ">
    <subfield code="a">it</subfield>
    <subfield code="a">en</subfield>
    <subfield code="a">fr</subfield>
</datafield>

WEKO 3

¶ 39 Leave a comment on paragraph 39 1 WEKO3 is a cloud-based repository system supported by JPCOAR (Japan Consortium for Open Access Repositories). It is developed based on INVENIO by CERN. In WEKO3, JPCOAR metadata schema is supported by default and a language attribute can be added for any metadata as long as it is allowed in the schema. Specifically, ISO-639-3 is acceptable as the language of the text and for a language attribute of other metadata elements, ISO-639-1 is acceptable. With each field, you can add a language tag in the form of a two-character ISO using the dropdown menu.

Comments

Susan Kung June 24, 2023 at 11:13 pm

This comment is actually for the text between examples 2 and 3, but that paragraph is not tagged for comments.
In general I agree with this, however, at The Archive of Indigenous Languages of Latin America (AILLA), we have found that this is not optimal in certain specific use cases dealing with educational materials or articles on “linguistic typology” or “linguistic diversity”, which frequently include a single or a few example(s) of a grammatical phenomenon in many different languages. We have found that when users are searching for content in a particular language, they get frustrated when they land on such examples in our repository because they either are not written in the language for which they are searching, or they do not have enough examples in or information about that language to be useful. This is, of course an edge case, but one to be considered. See e.g. Terrence Kaufman’s archived course on American Indian Languages https://ailla.utexas.org/islandora/object/ailla%3A137495.

2 Comments on paragraph 22

Kelly Stathis June 8, 2023 at 7:04 pm

This wording “Because of the cardinality [0-n]” is confusing, because xml:lang is a non-repeatable [0-1] attribute when applied to a specific element. I think the recommendation is clear from the below example – multiple languages should be in separate metadata fields, not combined – but this sentence almost suggests that xml:lang could be repeated within the same element.
Susan Kung June 24, 2023 at 11:16 pm

The guidelines are not clear on why the xml:lang attribute should be used instead of one more applicable to the metadata schema of the repository. The links are not helpful for sussing this out.
Here’s a MODS example from AILLA, where we use ISO 639-3 language codes:
<titleInfo lang=”eng”>
<title>Iskonawa Oral Tradition</title>
</titleInfo>
<titleInfo lang=”spa”>
<title>Tradición Oral Iskonawa</title>

</titleInfo>

0 Comments on paragraph 23

2 Comments on paragraph 24

François Renaville June 19, 2023 at 11:56 am

I would suggest recommending a specific order for core metadata fields like title in case there would be more than one title in the metadata, for example original title + title translated into English (or in any other language) for more visibility and discoverability.
I would suggest having the original title first, followed then by all additional/alternative titles. Thus, taking back your example:
<dc:title xml:lang=”fr”>Libre Accès</dc:title>
<dc:title xml:lang=”en”>Open Access</dc:title>
First dc:title with ‘fr’ language attribute because the describe record is in French and second dc:title with attribute ‘en’ because the submitter wants to make the record more visible and better harvested at an international level.

Use case:
Record https://hal.science/hal-03130990 describes a book in French. Main title is in French (Les grands discours à L’Unesco de 1945 à nos jours), but there is an additional title in English (Great speeches at Unesco from 1945 to nowadays) that can be displayed by switching from ‘fr’ to ‘en’. However, in the OAI output, the English title comes first: https://api.archives-ouvertes.fr/oai/hal/?verb=GetRecord&metadataPrefix=oai_dc&identifier=oai:HAL:hal-03130990v1 There is no guarantee that aggregators and harvesters will index and display both titles. This is a risk that only the first provided title is displayed (here English), while the describe record is in French. I have encountered this issue several times already.
JPCOAR(Japan Consortium for Open Access Repository) June 30, 2023 at 1:26 am

(This is actually a comment on Appendix 3.)There is a correction regarding the URL as follows.JPCOAR Metadata Schema 2.0 Drafthttps://schema.irdb.nii.ac.jp/ja/schema/2.0-draft/14https://schema.irdb.nii.ac.jp/ja/schema/2.0-draft/1 –>JPCOAR Metadata Schema 2.0https://schema.irdb.nii.ac.jp/en/schema/2.0/14https://schema.irdb.nii.ac.jp/en/schema/2.0/1

4 Comments on paragraph 25

François Renaville June 19, 2023 at 12:04 pm

I would avoid using the phrase “non-official languages”. What’s the point of being an official language or not? What about using rather “local”, “regional” or “less spread” languages? I would also be careful if you provide any language examples here (just to spare susceptibilities 😉 ).
François Renaville June 19, 2023 at 12:43 pm

Keep maybe also in mind that NISO ODI recommends on their side to always use the list of MARC language codes (https://www.loc.gov/marc/languages/language_code.html), thus ISO 639-2 (alpha-3 code).
NISO RP-19-2020, Open Discovery Initiative: Promoting Transparency in Discovery https://www.niso.org/publications/rp-19-2020-odi
Susan Kung June 24, 2023 at 11:17 pm

AILLA uses ISO 639-3 language codes, which are the most specific and usually the best option for Indigenous and minority languages. For languages that do not have an assigned 639-3 code, we assign a local code, but we are considering moving to assigning the code “mis” for Missing to these languages.
Joan Spanne June 27, 2023 at 8:26 pm

It is inappropriate to cite a long-obsolete RFC, RFC 1766, as an authority for this section. BCP 47 currently points to RFC 5646, the third RFC in sequence since 1766.https://datatracker.ietf.org/doc/html/rfc5646 or https://www.ietf.org/rfc/rfc5646.txtThe single letter ‘i’ subtag prefix has not been included in the RFC since 3066 (which followed 1766 and became obsolete in 2006), and is no longer in use for new subtags assigned by IANA. All 12 instances of IANA declared i-__ subtags were grandfathered into the current registry and 11 of the 12 have been deprecated since then. This approach should not be included as a recommendation.

0 Comments on paragraph 26

1 Comment on paragraph 27

Joan Spanne June 27, 2023 at 8:45 pm

In DSpace 7, the value-pairs set for languages can include whatever languages and language identifiers desired, as this is customized in the submission-forms.xml file. It can include three letter identifiers, if there are languages with three-letter identifiers that are present in the target material for a collection.
The paragraph’s middle sentences are not clearly worded.

0 Comments on paragraph 28

1 Comment on paragraph 29

Joan Spanne June 27, 2023 at 8:50 pm

It is odd to move between sections that are repository specific, then this section that is topic specific, then back to three more repository specific sections. Paragraphs 30, 31 here fit better in the DSpace and Dataverse sections, respectively.

François Renaville June 19, 2023 at 12:29 pm

As you say, Marc21 format uses ISO 639-2 language codes and I find it a bit odd that the group recommends using 369-1 encoding in MarcXML. This looks somehow contradictory to me. But if you do, I would suggest adding subfield “2” (Source of code) to be compliant with the Marc format. Thus, taking your example:

Possibility 1:
<datafield tag=”041″ ind1=” ” ind2=”7″>
<subfield code=”a”>it</subfield>
<subfield code=”a”>en</subfield>
<subfield code=”a”>fr</subfield>
<subfield code=”2″>iso639-1</subfield>
</datafield>

Possibility 2, since field 041 can be repeated:
<datafield tag=”041″ ind1=” ” ind2=”7″>
<subfield code=”a”>it</subfield>
<subfield code=”2″>iso639-1</subfield>
</datafield>
<datafield tag=”041″ ind1=” ” ind2=”7″>
<subfield code=”a”>en</subfield>
<subfield code=”2″>iso639-1</subfield>
</datafield>
<datafield tag=”041″ ind1=” ” ind2=”7″>
<subfield code=”a”>fr</subfield>
<subfield code=”2″>iso639-1</subfield>
</datafield>

Remarks:

Note that second indicator has been set to “7” (= Source specified in subfield $2)
First indicator set to “0” (as in your example) means that the item is not a translation or does not include any translation, which is somehow surprising with the three language codes you provide (although no impossible at all).
Marc control field 008 can only accept Marc codes (ISO 639-2) in positions 35-37. If only a non-MARC code is used in 041 to express the language in an item, field 008/35-37 should be coded with three fill characters (|||), not with the alpha-2 code!

1 Comment on paragraph 39

JPCOAR(Japan Consortium for Open Access Repository) June 30, 2023 at 1:16 am

I corrected the descriptions about WEKO3.before: WEKO3 is a cloud-based repository system supported by JPCOAR (Japan Consortium for Open Access Repositories). It is developed based on INVENIO by CERN.↓after: WEKO3 is a repository software developed by NII (National Institute of Informatics, Japan) based on INVENIO by CERN. This software operates JAIRO Cloud, a cloud-based repository system, which is supported by JPCOAR (Japan Consortium for Open Access Repositories) and NII.before: Specifically, ISO-639-3 is acceptable as the language of the text and for a language attribute of other metadata elements, ISO-639-1 is acceptable.↓after: Specifically, ISO-639-3 is acceptable as the language of the text, and for a language attribute of other metadata elements, ISO-639-1 is acceptable.before: With each field, you can add a language tag in the form of a two-character ISO using the dropdown menu.↓after: With WEKO3, you can add the language tag in the form of a two-character ISO using the dropdown menu, checkbox, and radio button.

Contents

Search

Special Pages

Table of Contents