Skip to content

Invalid multibyte string in parsed FAERS drug table #24

Description

@InfiniteStryker0

While building a FAERS 2016–2025 dataset and ran into what looks like a character encoding issue in the parsed drug table. After downloading/parsing FAERS data with the package, calling View(drug) in RStudio failed with:

Error in nchar(col, type = "width") : 
  invalid multibyte string, element 16213314

Inspecting the offending row showed that the issue appears to come from the lot_num field:

drug[16213314]

Output:

    year quarter primaryid   caseid drug_seq role_cod drugname  prod_ai val_vbm        route
   <int>  <char>    <char>   <char>    <int>   <char>   <char>   <char>   <int>       <char>
1:  2019      q3 165897511 16589751        1       PS  KINERET ANAKINRA       1 Subcutaneous

   dose_vbm cum_dose_chr cum_dose_unit dechal rechal      lot_num exp_dt nda_num dose_amt
     <char>        <num>        <char> <char> <char>       <char> <char>  <char>   <char>
1:     <NA>           NA          <NA>   <NA>      Y 33298?\xad1B   <NA>  103950      100

   dose_unit dose_form dose_freq
      <char>    <char>    <char>
1:        MG      <NA>        QD

The problematic value appears to be:

lot_num = "33298?\xad1B"

This seems to contain a malformed or non-UTF-8 byte sequence / soft hyphen-like character, which causes downstream display/string-width operations to fail.

Would it make sense for faers_parse() or the relevant table parsing helper to normalize character columns to valid UTF-8 before returning the parsed tables? This would help avoid failures in View(), nchar(type = "width"), and potentially downstream export to Parquet or other formats that expect valid UTF-8 strings.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions