Top suggestions for Language Models Generated Sequence LikelihoodExplore more searches like Language Models Generated Sequence LikelihoodPeople interested in Language Models Generated Sequence Likelihood also searched for |
+through+the+angle+of+time+directionality%252C+addressing+a+question+first+raised+in+(Shannon%252C+1951).+For+large+enough+models%252C+we+empirically+find+a+time+asymmetry+in+their+ability+to+learn+natural+language%3a+a+difference+in+the+average+log-perplexity+when+trying+to+predict+the+next+token+versus+when+trying+to+predict+the+previous+one.+This+difference+is+at+the+same+time+subtle+and+very+consistent+across+various+modalities+(language%252C+model+size%252C+training+time%252C+...).+Theoretically%252C+this+is+surprising%3a+from+an+information-theoretic+point+of+view%252C+there+should+be+no+such+difference.+We+provide+a+theoretical+framework+to+explain+how+such+an+asymmetry+can+appear+from+sparsity+and+computational+complexity+considerations%252C+and+outline+a+number+of+perspectives+opened+by+our+results.&cdnurl=https%3a%2f%2fth.bing.com%2fth%2fid%2fR.32684ea38c37e0dd51eb2262155ac8a9%3frik%3d4UU3XVAVejAa%252bg%26pid%3dImgRaw%26r%3d0&exph=630&expw=1200&q=Language+Models+Generated+Sequence+Likelihood&simid=608039474679856041&FORM=IRPRST&ck=E80A7B7A2DDE85D9358582D3624DD46E&selectedIndex=8&itb=0)
+memorize+text+from+several+sources+of+documents.+In+pretraining%252C+LLM+trains+to+maximize+the+likelihood+of+text+but+neither+receives+the+source+of+the+text+nor+memorizes+the+source.+Accordingly%252C+LLM+can+not+provide+document+information+on+the+generated+content%252C+and+users+do+not+obtain+any+hint+of+reliability%252C+which+is+crucial+for+factuality+or+privacy+infringement.+This+work+introduces+token-level+source+identification+in+the+decoding+step%252C+which+maps+the+token+representation+to+the+reference+document.+We+propose+a+bi-gram+source+identifier%252C+a+multi-layer+perceptron+with+two+successive+token+representations+as+input+for+better+generalization.+We+conduct+extensive+experiments+on+Wikipedia+and+PG19+datasets+with+several+LLMs%252C+layer+locations%252C+and+identifier+sizes.+The+overall+results+show+a+possibility+of+token-level+source+identifiers+for+tracing+the+document%252C+a+crucial+problem+for+the+safe+use+of+LLMs.&cdnurl=https%3a%2f%2fth.bing.com%2fth%2fid%2fR.f2b6f209ca48816108bafe47098897c6%3frik%3dsUEfKU7CKASbPQ%26pid%3dImgRaw%26r%3d0&exph=630&expw=1200&q=Language+Models+Generated+Sequence+Likelihood&simid=608008980441598361&FORM=IRPRST&ck=254F45BF74A129AAF50BE443B273A808&selectedIndex=10&itb=0)
Explore more searches like Language Models
Generated Sequence Likelihood
Some results have been hidden because they may be inaccessible to you.Show inaccessible results
Feedback
