Opening the Repository

[Ovid]

@druud @happy-barney @perigrin @rabbiveesh @rwp0 @SysPete @tobyink @demerphq

Just a heads up to everyone: before my keynote in Toronto, I want to open this repository to the public and gain some traction.

Also, I've considered renaming it to Ọrunmila. He's a spirit of wisdom from the Yoruba religion. I'd like projects to be more inclusive than simply the white European traditions.

[tobyink]

Ọrunmila seems like a good name, but maybe ASCIIfy it just in case it breaks tools, and also to make it easier for people to type?

[perigrin]

Something something inclusion vs appropriation … I’m gonna shut the hell up about naming things cause I wrote a module called Kensho.

I have no issues with opening things up.

[happy-barney]

honestly, I don't care about project name, I care about functionality (especially if it has potential to put Perl eventually ahead of train instead of catching it)

[rwp0]

@druud @happy-barney @perigrin @rabbiveesh @rwp0 @SysPete @tobyink @demerphq

Just a heads up to everyone: before my keynote in Toronto, I want to open this repository to the public and gain some traction.

Also, I've considered renaming it to Ọrunmila. He's a spirit of wisdom from the Yoruba religion. I'd like projects to be more inclusive than simply the white European traditions.

Ok but what wisdom has to do with type checks specifically?

I liked the way you explained Heimdall's role in guarding against invasions (ie. bugs but actually uninvented guests that is unexpected function arguments and troubles invented by passing them to unintended functions) 🐛

But yeah I agree that Heimdall is overused and you should try to be more inclusive targeting other world cultures as well.

Maybe find other Yoruba (or another culture's) deity which be analogous to Heimdall's role in guarding and would match the concept of type constraints better?

[perigrin]

The system of divination used in Ifá is a code to access the scientific and metaphysical knowledge in the literary corpus; the Odu Ifá. Orunmila is identified as the Grand Priest, as he revealed (read; invented) the foundational divinity and prophecy (the first 16 Odu) to the world.

If I squint Ifá sounds like a systematic approach to annotating metaphysics and science. Like say a system of constraints do for a software program. If I squint.

Not all cultures have easily overlapping concepts.

[happy-barney]

Ok but what wisdom has to do with type checks specifically?

Don't use word type, use data contract - and here you are, contract implies wisdom :-)

[Ovid]

@happy-barney Yes, we're avoiding the word "type" because it's so overloaded. However, we're going with checks becauase:

• Huffman encoding
• There are potential issues with complex structures, so "contract" may not be accurate

As for the latter point, I don't want to write checks we can't cash (as we say in Texas)  😃

[happy-barney]

I created discussion earlier regarding naming - #18

re huffman encoding:

• advantage of having single keyword for all usages imho beats it

re complex structurs

• data contract is same as check here

Plus additional notes:

• nasty English here:

  • check can be verb and noun both having close meaning: check check

• contract is commonly used in books (as data contract) - having such name will be imho better in selling Perl, for example something like:
  Perl supports native data contracts, unlike other languages which uses different surrogates.

[Ovid]

Your comments on the "nasty" English and "better in selling Perl" are both spot on. I really like that, but "contract" has implications. Let me quote from something Damian wrote on matter. If we accept his reasoning, "contract" is the wrong word.

---

Why we can’t have nice things

Part 2: Semantics

I want to point out an aspect of the current Heimdall design that concerns me...but for which I don’t have a good solution.

There’s no problem when I specify:

    my $result :of(INT);
    my %grade  :of(INT);
    my @scores :of(INT);

Any modification I subsequently make to the value(s) in $result, %grades, or @scores is simply checked to verify that the variable contains only integers (and, yeah, there are a surprisingly large number of ways to assign an invalid value into an array):

    $result = $value;                  # Exception if $value isn't an integer
    undef $result;                     # Exception, because undef isn't an integer

    $grade{k} = $value;                # Exception if $value isn't an integer
    %grade    = (k => $value);         # Exception if $value isn't an integer
    undef $grade{k};                   # Exception, because undef isn't an integer

    $scores[0] = $value;               # Exception if $value isn't an integer
    @scores    = $value;               # Exception if $value isn't an integer
    $#scores   = $new_length;          # Exception if new length adds undef elements
    undef   $scores[0];                # Exception, because undef isn't an integer
    delete  $scores[0];                # Exception if the deletion leaves an undef element
    push    @scores, $value;           # Exception if $value isn't an integer
    unshift @scores, $value;           # Exception if $value isn't an integer
    splice  @scores, 0, 1, $value;     # Exception if $value isn't an integer

That means that, despite the manifold ways of assigning a non-integer into a variable, the :of(INT) check is a reliable guarantee that your code won’t do so without your knowledge.

Great. That’s precisely why we wanted checks in the first place. As a guarantee that the value in a variable has the properties we want/expect/need.

But there’s a problem with more complex checks. Specifically, with any check that involves a reference. For example:

    my $result :of(HASH[INT]);
    my %grade  :of(TUPLE[STR,INT]);
    my @scores :of(ARRAY[INT]);

These variables still successfully catch any attempt to directly assign them an incompatible value:

    $result = $value;                  # Exception if $value isn't a reference to a hash of integers
    undef $result;                     # Exception, because undef isn't a reference to a hash of integers

    $grade{k} = $value;                # Exception if $value isn't a reference to a two-element array
    %grade    = (k => $value);         # Exception if $value isn't a reference to a two-element array integer
    undef $grade{k};                   # Exception, because undef isn't a reference to an two-element array

    $scores[0] = $value;               # Exception if $value isn't a reference to an array of integers
    @scores    = $value;               # Exception if $value isn't a reference to an array of integers
    $#scores   = $new_length;          # Exception if new length adds undef elements
    undef   $scores[0];                # Exception, because undef isn't a reference to an array of integers
    delete  $scores[0];                # Exception if the deletion leaves an undef element
    push    @scores, $value;           # Exception if $value isn't a reference to an array of integers
    unshift @scores, $value;           # Exception if $value isn't a reference to an array of integers
    splice  @scores, 0, 1, $value;     # Exception if $value isn't a reference to an array of integers

But in the current design, the checks are only applied to direct assignments made to the variables themselves. Assignments into some subelement of the variables are not checked.

For example:

    $result->{k} = 'non-int';               # NO exception (because $result itself is not being modified)
    undef $result->{k};                     # NO exception (because $result itself is not being modified)

    $grade{k}[1] = 'non-int';               # NO exception (because %grade itself is not being modified)
    undef $grade{k}[1];                     # NO exception (because %grade itself is not being modified)

    $scores[0][0]   = 'non-int';            # NO exception (because @scores itself is not being modified)
    $scores[0]->@*  = 'non-int';            # NO exception (because @scores itself is not being modified)
    $scores[0]->$#* = $new_length;          # NO exception (because @scores itself is not being modified)
    undef   $scores[0][0];                  # NO exception (because @scores itself is not being modified)
    delete  $scores[0][0];                  # NO exception (because @scores itself is not being modified)
    push    $scores[0]->@*, 'non-int';      # NO exception (because @scores itself is not being modified)
    unshift $scores[0]->@*, 'non-int';      # NO exception (because @scores itself is not being modified)
    splice  $scores[0]->@*, 0,1,'non-int';  # NO exception (because @scores itself is not being modified)

This means that a check applied to a variable is not at all cases a guarantee about the value(s) stored in that variable; only a guarantee that no invalid value(s) will ever be directly assigned to that variable.

And my problem with that is that I don’t know if that’s a problem.

From one point of view, I don’t feel it is. If we clearly state that checks are not a type system, but only validate direct modifications to the checked variable, and if we specifically document the indirect modifications that checks don’t cover, then we can set reasonable expectations for the capacities and the limitations of the proposed mechanism.

On the other hand, some people at least are surely going to expect (and not unreasonably) that if they specify:

    my $result :of(HASH[INT]);

...then $result will always contain a reference to a hash whose values are always integers. And the current proposal does not – and cannot – meet that expectation.

Of course, we could change the check mechanism, so that (for example) when a hashref is assigned to $result, it would automagically be endowed with its own check (i.e. :of(INT)), consistent with the overall check applied to $result. Then the second-level contents of $result would be guaranteed to be integers, because any indirect assignment into the second layer would have to pass the new :of(INT) check. So the overarching :of(HASH[INT]) on $result would be an actual guarantee, even when $result is itself unmodified by an indirect operation.

But that seems...problematical. One of the key design goals here is to make checks as light-weight and as efficient as possible. But recursively applying new checks (at runtime) all the way through a nested data structure would necessarily increase the cost of any access into that structure.

Moreover, magically imposing additional subchecks on nested components would increase the opportunities for unintended and undocumented coupling between variables, opening up the possibility of spooky action at a distance. For example:

    my @data = 1..99;              # Note: no check on this array

    # And later...

    my @scores :of(ARRAY[INT]);

    # And later again...

    $scores[0] = \@data;           # @data is implicitly endowed with an :of(INT) check

    # And later still...

    $data[0] = 'fail';             # KABOOM! Value 'fail' does not pass the check on @data
                                   # (What check on @data??? There's no check on the declaration of @data! WTF???)

So I don’t think autovivifying checks on nested data structures would be a viable solution here. In fact, I don’t think there is a viable solution here. It’s not a technical problem, it’s a human problem: problem of expectations. A clash between the freedom and flexibility offered by references, and the security and assurances desired by developers.

Of course, that’s why they’re called “checks”, and not “types”. They’re explicitly not guarantees about the invariants of a data structure, they’re just assurances that a data structure cannot be directly modified in undesirable ways. If that’s made sufficiently clear, then the design is reasonable and appropriate.

But can it be made sufficiently clear? And can the expectations of end-users be kept reasonable? It worries me.

BTW, none of this applies to “API” checks: those applied to a subroutine’s parameters or return values. A parameter variable is passed its value in the equivalent of a simple assignment, so the parameter can directly check the entire nested datastructure it is being passed at that point. Likewise a return value is effectively a constant at the point it is actually being returned, so that value can be checked entirely for conformity to any check.

Of course, Perl parameters are also read-write within their subroutine body (a mistake in my view, but it’s far too late to do anything about that now), so the same problems do subsequently apply to checked parameter variables whose values are indirectly modified within their subroutine. In other words, checked parameters are guaranteed to be passed the correct data structure, but that data structure is not guaranteed to stay correct if the subroutine code subsequently performs indirect modifications to the nested data through the parameter variable.

In other words, the current design of checks is perfectly safe and reliable for purely functional code and for simple direct assignments. But not for indirect assignments via references.

And I honesty don’t know whether that’s a limitation, an issue, a problem, or a disaster.

[rwp0]

Ok but what wisdom has to do with type checks specifically?

Don't use word type, use data contract - and here you are, contract implies wisdom :-)

Ok, but that's not plain, but rather a concealed and esoteric meaning requiring somewhat deep insight into the matter 🤔

Let's try keeping everything esp. the names simple without further derivations 🙂

[Ovid]

I think @rwp0 made a good point about the proposed name. I wasn't quite convinced with Ifá, but now I'm looking at Oshun.

• Oshun is the protector, saviour, or nurturer of humanity. Maintains spiritual balance.
• ASCII names are less likely to cause issues (but like "Corinna", it's temporary)

[Ovid]

FYI: the repository is now public. I'll announce it tomorrow at the conference.