Skip to content

Since bincode is unmaintained use wincode instead. #646

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ltratt merged 3 commits into from
Jul 17, 2026
Merged

Since bincode is unmaintained use wincode instead. #646

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
36d77a5
Fix typo, and test in cttests
ratmice Jul 16, 2026
0109e96
Since bincode is unmaintained use wincode instead.
ratmice Jul 16, 2026
2a64400
Make serialisation format configurable
ratmice Jul 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,22 +28,22 @@ codegen-units = 1
panic = 'unwind'

[workspace.dependencies]
bincode = "2.0"
wincode = "0.5.5"
cactus = "1.0"
filetime = "0.2"
fnv = "1.0"
getopts = "0.2"
glob = "0.3"
indexmap = "2"
num-traits = "0.2"
packedvec = "1.2"
packedvec = "2.0"
quote = "1.0"
regex = "1.3"
regex-syntax = "0.8"
serde = "1.0"
sparsevec = "0.2.2"
sparsevec = "0.3.0"
unicode-width = "0.1.11"
vob = "3.0.4"
vob = "4.0.0"
proc-macro2 = "1.0"
prettyplease = "0.2.31"
syn = "2.0"
Expand Down
4 changes: 2 additions & 2 deletions cfgrammar/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,14 @@ keywords = ["yacc", "grammar"]

[features]
serde = ["dep:serde", "serde/derive", "vob/serde"]
bincode = ["dep:bincode", "vob/bincode"]
wincode = ["dep:wincode", "vob/wincode"]

[lib]
name = "cfgrammar"
path = "src/lib/mod.rs"

[dependencies]
bincode = { workspace = true, optional = true, features = ["derive"] }
wincode = { workspace = true, optional = true, features = ["derive"] }
indexmap.workspace = true
num-traits.workspace = true
regex.workspace = true
Expand Down
59 changes: 30 additions & 29 deletions cfgrammar/src/lib/header.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,32 @@ impl From<Value<Span>> for Value<Location> {
}
}

impl<T> Value<T> {
pub fn primary_location(&self) -> &T {

@ratmice ratmice Jul 16, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a pretty awkward API, the thing going on is that we've parameterized between
logical locations (command line arguments in nimbleparse, builder parameters in CTParserBuilder, and source locations read from %grmtools sections).

In the first two cases we have Value<Span> while in the second we have Value<Location>.

We could return a Vec<T> where T: Clone here, but in all the Value<Location> areas, this is going to be extremely unhelpful, and in the Span case, PrimaryLocation will still return at least one relevant location.

So we may drop some information regarding source locations when dealing with Span,
but we also avoid emitting redundant locations, when all those locations are like CommandLine

@ratmice ratmice Jul 16, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, it's kind of like we're either going to have this information loss, where we drop "secondary" locations,
or we're going to have a bunch of redundancy. So I think picking one of the locations is ok.

@ltratt ltratt Jul 16, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we've got a major version bump coming up, we could consider an API change if it would help?

@ratmice ratmice Jul 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hadn't given it a whole lot of thought, In that the whole mixing of spans and logical error locations is kind of new territory for me, so I've kind of been winging it. One thing we could consider is the way that the AST Value<T> is currently an enum which has the pair of e.g. String, and Span, but also String, and Location.

We could consider making it a

enum Setting {
   Num(u64, usize)
   Flag(bool, usize),
   Array(Vec<Setting>, usize, usize)
}
struct ValueRepr<T> {
    value: Value,
    locations: Vec<T>   
}

With that the Location::CommandLine and Location::Other("From<...>") could just push one location on the locations stack, and we'd use the usize indexes to point to that one location.

With spans, each of these could refer to different locations on the locations stack.

I think one of the questions is whether we could get the lrpar/cttests/grmtools_section.test to emit that structure.
Because we test that the AST structure is exactly the same for the handwritten parser, and the generated parser in the testsuite. It's hard for me to imagine how this structure representation could be emitted from the generated parser's rules. I'll try and give it a think overnight.

@ltratt ltratt Jul 16, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, perhaps the current muddle is the best we can do right now.

@ratmice ratmice Jul 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, I did think of one more place that could be cleaned up in e8a5dab, Edit: the other thing to note is that this is all #[doc(hidden)] too.

@ltratt ltratt Jul 17, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point!

match self {
Value::Flag(_, loc) => loc,
Value::Setting(setting) => setting.primary_location(),
}
}
}

impl<T> Setting<T> {
fn primary_location(&self) -> &T {
match self {
Self::Constructor { arg, .. } => arg.primary_location(),
Self::Unitary(ns) => ns.primary_location(),
Self::Array(_, start_loc, _) => start_loc,
Self::Num(_, loc) | Self::String(_, loc) => loc,
}
}
}

impl<T> Namespaced<T> {
fn primary_location(&self) -> &T {
&self.member.1
}
}

static RE_LEADING_WS: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"^[\p{Pattern_White_Space}]*").unwrap());
static RE_NAME: LazyLock<Regex> = LazyLock::new(|| {
Expand Down Expand Up @@ -561,35 +587,6 @@ impl<T: Clone> TryFrom<&Value<T>> for YaccKind {
fn try_from(value: &Value<T>) -> Result<YaccKind, HeaderError<T>> {
let mut err_locs = Vec::new();
match value {
Value::Flag(_, loc) => Err(HeaderError {
kind: HeaderErrorKind::ConversionError(
"From<YaccKind>",
"Cannot convert boolean to YaccKind",
),
locations: vec![loc.clone()],
}),
Value::Setting(Setting::Num(_, loc)) => Err(HeaderError {
kind: HeaderErrorKind::ConversionError(
"From<YaccKind>",
"Cannot convert number to YaccKind",
),
locations: vec![loc.clone()],
}),
Value::Setting(Setting::Array(_, loc, _)) => Err(HeaderError {
kind: HeaderErrorKind::ConversionError(
"From<YaccKind>",
"Cannot convert array to YaccKind",
),
locations: vec![loc.clone()],
}),
Value::Setting(Setting::String(_, loc)) => Err(HeaderError {
kind: HeaderErrorKind::ConversionError(
"From<YaccKind>",
"Cannot convert string to YaccKind",
),
locations: vec![loc.clone()],
}),

Value::Setting(Setting::Unitary(Namespaced {
namespace,
member: (yk_value, yk_value_loc),
Expand Down Expand Up @@ -676,6 +673,10 @@ impl<T: Clone> TryFrom<&Value<T>> for YaccKind {
})
}
}
val => Err(HeaderError {
kind: HeaderErrorKind::InvalidEntry("yacckind"),
locations: vec![val.primary_location().clone()],
}),
}
}
}
Expand Down
6 changes: 3 additions & 3 deletions cfgrammar/src/lib/idxnewtype.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,18 @@

use std::mem::size_of;

#[cfg(feature = "bincode")]
use bincode::{Decode, Encode};
use num_traits::{PrimInt, Unsigned};
#[cfg(feature = "serde")]
use serde::{Deserialize, Serialize};
#[cfg(feature = "wincode")]
use wincode::{SchemaRead, SchemaWrite};

macro_rules! IdxNewtype {
($(#[$attr:meta])* $n: ident) => {
$(#[$attr])*
#[derive(Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]
#[cfg_attr(feature="serde", derive(Serialize, Deserialize))]
#[cfg_attr(feature="bincode", derive(Encode, Decode))]
#[cfg_attr(feature="wincode", derive(SchemaRead, SchemaWrite))]
pub struct $n<T>(pub T);

impl<T: PrimInt + Unsigned> From<$n<T>> for usize {
Expand Down
6 changes: 3 additions & 3 deletions cfgrammar/src/lib/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,10 +51,10 @@
//! [`YaccGrammar::new_with_storaget()`](yacc/grammar/struct.YaccGrammar.html#method.new_with_storaget)
//! which take as input a Yacc grammar.

#[cfg(feature = "bincode")]
use bincode::{Decode, Encode};
#[cfg(feature = "serde")]
use serde::{Deserialize, Serialize};
#[cfg(feature = "wincode")]
use wincode::{SchemaRead, SchemaWrite};

#[doc(hidden)]
pub mod header;
Expand All @@ -73,7 +73,7 @@ pub use crate::idxnewtype::{PIdx, RIdx, SIdx, TIdx};

#[derive(Clone, Copy, Debug, Hash, Eq, PartialEq)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "bincode", derive(Encode, Decode))]
#[cfg_attr(feature = "wincode", derive(SchemaRead, SchemaWrite))]
pub enum Symbol<StorageT> {
Rule(RIdx<StorageT>),
Token(TIdx<StorageT>),
Expand Down
6 changes: 3 additions & 3 deletions cfgrammar/src/lib/span.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
#[cfg(feature = "bincode")]
use bincode::{Decode, Encode};
use proc_macro2::TokenStream;
use quote::{ToTokens, TokenStreamExt, quote};
#[cfg(feature = "serde")]
use serde::{Deserialize, Serialize};
#[cfg(feature = "wincode")]
use wincode::{SchemaRead, SchemaWrite};

/// A `Span` records what portion of the user's input something (e.g. a lexeme or production)
/// references (i.e. the `Span` doesn't hold a reference / copy of the actual input).
#[derive(Clone, Copy, Debug, Eq, PartialEq, Hash)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "bincode", derive(Encode, Decode))]
#[cfg_attr(feature = "wincode", derive(SchemaRead, SchemaWrite))]
pub struct Span {
start: usize,
end: usize,
Expand Down
99 changes: 5 additions & 94 deletions cfgrammar/src/lib/yacc/grammar.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
#![allow(clippy::derive_partial_eq_without_eq)]
use std::{cell::RefCell, collections::HashMap, fmt::Write, str::FromStr};

#[cfg(feature = "bincode")]
use bincode::{Decode, Encode};
use num_traits::{AsPrimitive, PrimInt, Unsigned};
#[cfg(feature = "serde")]
use serde::{Deserialize, Serialize};
use vob::Vob;
#[cfg(feature = "wincode")]
use wincode::{SchemaRead, SchemaWrite};

use super::{
YaccKind, ast,
Expand All @@ -23,15 +23,15 @@ const IMPLICIT_START_RULE: &str = "^~";
pub type PrecedenceLevel = u64;
#[derive(Clone, Copy, Debug, PartialEq)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "bincode", derive(Encode, Decode))]
#[cfg_attr(feature = "wincode", derive(SchemaRead, SchemaWrite))]
pub struct Precedence {
pub level: PrecedenceLevel,
pub kind: AssocKind,
}

#[derive(Clone, Copy, Debug, PartialEq)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "bincode", derive(Encode, Decode))]
#[cfg_attr(feature = "wincode", derive(SchemaRead, SchemaWrite))]
pub enum AssocKind {
Left,
Right,
Expand All @@ -41,7 +41,7 @@ pub enum AssocKind {
/// Representation of a `YaccGrammar`. See the [top-level documentation](../../index.html) for the
/// guarantees this struct makes about rules, tokens, productions, and symbols.
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "bincode", derive(Encode))]
#[cfg_attr(feature = "wincode", derive(SchemaRead, SchemaWrite))]
pub struct YaccGrammar<StorageT = u32> {
/// How many rules does this grammar have?
rules_len: RIdx<StorageT>,
Expand Down Expand Up @@ -103,97 +103,8 @@ pub struct YaccGrammar<StorageT = u32> {
expectrr: Option<usize>,
}

// The implementations of `Decode` and `BorrowDecode`
// Contain a modified copy of the output of `cargo expand` of the derivation:
// #[cfg_attr(feature = "bincode", derive(Decode))]
//
// The current version of bincode has bugs in it's derive macro implementation
// https://github.com/bincode-org/bincode/issues/763
// https://github.com/bincode-org/bincode/issues/646
//
// Once those are fixed, we can replace this with derive.

#[cfg(feature = "bincode")]
impl<StorageT, __Context> Decode<__Context> for YaccGrammar<StorageT>
where
StorageT: Decode<__Context> + 'static,
{
fn decode<__D: bincode::de::Decoder<Context = __Context>>(
decoder: &mut __D,
) -> Result<Self, bincode::error::DecodeError> {
Ok(Self {
rules_len: Decode::decode(decoder)?,
rule_names: Decode::decode(decoder)?,
token_names: Decode::decode(decoder)?,
token_precs: Decode::decode(decoder)?,
token_epp: Decode::decode(decoder)?,
tokens_len: Decode::decode(decoder)?,
eof_token_idx: Decode::decode(decoder)?,
prods_len: Decode::decode(decoder)?,
start_prod: Decode::decode(decoder)?,
prods: Decode::decode(decoder)?,
rules_prods: Decode::decode(decoder)?,
prods_rules: Decode::decode(decoder)?,
prod_precs: Decode::decode(decoder)?,
prod_spans: Decode::decode(decoder)?,
implicit_rule: Decode::decode(decoder)?,
actions: Decode::decode(decoder)?,
action_spans: Decode::decode(decoder)?,
parse_param: Decode::decode(decoder)?,
parse_generics: Decode::decode(decoder)?,
programs: Decode::decode(decoder)?,
actiontypes: Decode::decode(decoder)?,
avoid_insert: Decode::decode(decoder)?,
expect: Decode::decode(decoder)?,
expectrr: Decode::decode(decoder)?,
})
}
}

// Eventually this should just be provided through:
// #[cfg_attr(feature = "bincode", derive(Decode))]
//
// See comment at the corresponding bincode::Decode impl.
#[cfg(feature = "bincode")]
impl<'__de, StorageT, __Context> bincode::BorrowDecode<'__de, __Context> for YaccGrammar<StorageT>
where
StorageT: bincode::de::BorrowDecode<'__de, __Context> + '__de,
{
fn borrow_decode<__D: ::bincode::de::BorrowDecoder<'__de, Context = __Context>>(
decoder: &mut __D,
) -> Result<Self, bincode::error::DecodeError> {
Ok(Self {
rules_len: bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
rule_names: ::bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
token_names: ::bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
token_precs: ::bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
token_epp: ::bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
tokens_len: ::bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
eof_token_idx: ::bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
prods_len: ::bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
start_prod: ::bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
prods: ::bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
rules_prods: ::bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
prods_rules: ::bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
prod_precs: ::bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
prod_spans: ::bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
implicit_rule: ::bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
actions: ::bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
action_spans: ::bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
parse_param: ::bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
parse_generics: ::bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
programs: ::bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
actiontypes: ::bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
avoid_insert: ::bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
expect: ::bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
expectrr: ::bincode::BorrowDecode::<'_, __Context>::borrow_decode(decoder)?,
})
}
}

// Internally, we assume that a grammar's start rule has a single production. Since we manually
// create the start rule ourselves (without relying on user input), this is a safe assumption.

impl YaccGrammar<u32> {
pub fn new(yk: YaccKind, s: &str) -> YaccGrammarResult<Self> {
YaccGrammar::new_with_storaget(yk, s)
Expand Down
8 changes: 4 additions & 4 deletions cfgrammar/src/lib/yacc/parser.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
// Note: this is the parser for both YaccKind::Original(YaccOriginalActionKind::GenericParseTree) and YaccKind::Eco yacc kinds.

#[cfg(feature = "bincode")]
use bincode::{Decode, Encode};
use num_traits::PrimInt;
use regex::Regex;
#[cfg(feature = "serde")]
Expand All @@ -13,6 +11,8 @@ use std::{
str::FromStr,
sync::LazyLock,
};
#[cfg(feature = "wincode")]
use wincode::{SchemaRead, SchemaWrite};

use crate::{
Span, Spanned,
Expand Down Expand Up @@ -161,7 +161,7 @@ impl fmt::Display for YaccGrammarErrorKind {
/// The various different possible Yacc parser errors.
#[derive(Debug, PartialEq, Eq, Clone)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "bincode", derive(Encode, Decode))]
#[cfg_attr(feature = "wincode", derive(SchemaRead, SchemaWrite))]
#[non_exhaustive]
pub enum YaccGrammarWarningKind {
UnusedRule,
Expand All @@ -171,7 +171,7 @@ pub enum YaccGrammarWarningKind {
/// Any Warning from the Yacc parser returns an instance of this struct.
#[derive(Debug, PartialEq, Eq, Clone)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "bincode", derive(Encode, Decode))]
#[cfg_attr(feature = "wincode", derive(SchemaRead, SchemaWrite))]
pub struct YaccGrammarWarning {
/// The specific kind of warning.
pub(crate) kind: YaccGrammarWarningKind,
Expand Down
2 changes: 2 additions & 0 deletions doc/src/yaccextensions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,11 @@ But a default can be set or forced by using a `YaccKindResolver`.
| `yacckind` | [YaccKind](yacccompatibility.md#yacckinds) | &checkmark; |
| `recoverykind` | [RecoveryKind](errorrecovery.md#recoverykinds) | &cross; |
| `test_files`[^†] | Array of string values | &cross; |
| `serialisation_format`[^⹋] | `lrpar::SerialisationFormat` | &cross; |

[^†]: Strings containing globs are resolved relative to the yacc `.y` source file.
`test_files` is currently experimental.
[^⹋]: Modifies the encoding algorithm used to serialise parser data in the generated code.

## Example

Expand Down
2 changes: 1 addition & 1 deletion lrlex/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ regex-syntax.workspace = true
num-traits.workspace = true
proc-macro2.workspace = true
quote.workspace = true
bincode.workspace = true
wincode.workspace = true
serde = { workspace = true, optional = true }
prettyplease.workspace = true
syn.workspace = true
Expand Down
Loading